Alluxio Write Location Policy - (SpecificHostPolicy)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Alluxio Write Location Policy - (SpecificHostPolicy)

Md Mahbub Alam
Hello,

does anyone have any experience in using SpecificHostPolicy(alluxio.client.file.policy.SpecificHostPolicy) to write data into Alluxio in Multi-Node cluster? Currently, I am using Round-Robin Policy. I need to write a specific file (test.csv) to a specific host. I configured the Alluxio with SpecificHostPolicy but how do I select specific host each time I run the program or copy the data into Alluxio. I will appreciate any kind of suggestion.

thanks
Mahbub

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio Write Location Policy - (SpecificHostPolicy)

Andrew Audibert
Hi Mahbub,

Unfortunately there isn't currently a way to specify the host through configuration properties, so SpecificHostPolicy is only usable through the java API where you can instantiate a new SpecificHostPolicy(hostname) and pass it as a method option. You can do that with

FileSystem fs = FileSystem.Factory.get();
OutputStream outStream = fs.createFile(new AlluxioURI("test.csv"), CreateFileOptions.defaults().setLocationPolicy(new SpecificHostPolicy("specific_hostname")));
// write data
outStream.close();

- Andrew

On Mon, Jun 25, 2018 at 7:54 AM Md Mahbub Alam <[hidden email]> wrote:
Hello,

does anyone have any experience in using SpecificHostPolicy(alluxio.client.file.policy.SpecificHostPolicy) to write data into Alluxio in Multi-Node cluster? Currently, I am using Round-Robin Policy. I need to write a specific file (test.csv) to a specific host. I configured the Alluxio with SpecificHostPolicy but how do I select specific host each time I run the program or copy the data into Alluxio. I will appreciate any kind of suggestion.

thanks
Mahbub

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio Write Location Policy - (SpecificHostPolicy)

Md Mahbub Alam
Hi Andrew,

Thanks. I was able to write/read alluxio using SpecificHostPolicy. I tried to check the data using "Alluxio fs ls /" and web interface, it shows the data same way in all nodes. I just want to know, is there any way to know (or see) which data file belongs to which host?

thanks again
Mahbub

On Monday, June 25, 2018 at 5:30:52 PM UTC-3, Andrew Audibert wrote:
Hi Mahbub,

Unfortunately there isn't currently a way to specify the host through configuration properties, so SpecificHostPolicy is only usable through the java API where you can instantiate a new SpecificHostPolicy(hostname) and pass it as a method option. You can do that with

FileSystem fs = FileSystem.Factory.get();
OutputStream outStream = fs.createFile(new AlluxioURI("test.csv"), CreateFileOptions.defaults().setLocationPolicy(new SpecificHostPolicy("specific_hostname")));
// write data
outStream.close();

- Andrew

On Mon, Jun 25, 2018 at 7:54 AM Md Mahbub Alam <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="cHMXNIOJBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">emahb...@...> wrote:
Hello,

does anyone have any experience in using SpecificHostPolicy(alluxio.client.file.policy.SpecificHostPolicy) to write data into Alluxio in Multi-Node cluster? Currently, I am using Round-Robin Policy. I need to write a specific file (test.csv) to a specific host. I configured the Alluxio with SpecificHostPolicy but how do I select specific host each time I run the program or copy the data into Alluxio. I will appreciate any kind of suggestion.

thanks
Mahbub

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="cHMXNIOJBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio Write Location Policy - (SpecificHostPolicy)

Andrew Audibert
Hi Mahbub,

If you go to the Browse view in the UI and click a file, it will show a list of locations where each block of the file is stored. You can also get this information with the "stat" CLI command:

bin/alluxio fs stat /path/to/file

To get block location information through the Java API, use the getStatus() command.

- Andrew

On Mon, Jun 25, 2018 at 5:39 PM Md Mahbub Alam <[hidden email]> wrote:
Hi Andrew,

Thanks. I was able to write/read alluxio using SpecificHostPolicy. I tried to check the data using "Alluxio fs ls /" and web interface, it shows the data same way in all nodes. I just want to know, is there any way to know (or see) which data file belongs to which host?

thanks again
Mahbub

On Monday, June 25, 2018 at 5:30:52 PM UTC-3, Andrew Audibert wrote:
Hi Mahbub,

Unfortunately there isn't currently a way to specify the host through configuration properties, so SpecificHostPolicy is only usable through the java API where you can instantiate a new SpecificHostPolicy(hostname) and pass it as a method option. You can do that with

FileSystem fs = FileSystem.Factory.get();
OutputStream outStream = fs.createFile(new AlluxioURI("test.csv"), CreateFileOptions.defaults().setLocationPolicy(new SpecificHostPolicy("specific_hostname")));
// write data
outStream.close();

- Andrew

On Mon, Jun 25, 2018 at 7:54 AM Md Mahbub Alam <[hidden email]> wrote:
Hello,

does anyone have any experience in using SpecificHostPolicy(alluxio.client.file.policy.SpecificHostPolicy) to write data into Alluxio in Multi-Node cluster? Currently, I am using Round-Robin Policy. I need to write a specific file (test.csv) to a specific host. I configured the Alluxio with SpecificHostPolicy but how do I select specific host each time I run the program or copy the data into Alluxio. I will appreciate any kind of suggestion.

thanks
Mahbub

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio Write Location Policy - (SpecificHostPolicy)

Md Mahbub Alam
thanks a lot, Andrew. 

On Monday, June 25, 2018 at 10:11:04 PM UTC-3, Andrew Audibert wrote:
Hi Mahbub,

If you go to the Browse view in the UI and click a file, it will show a list of locations where each block of the file is stored. You can also get this information with the "stat" CLI command:

bin/alluxio fs stat /path/to/file

To get block location information through the Java API, use the getStatus() command.

- Andrew

On Mon, Jun 25, 2018 at 5:39 PM Md Mahbub Alam <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="v4dZss2YBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">emahb...@...> wrote:
Hi Andrew,

Thanks. I was able to write/read alluxio using SpecificHostPolicy. I tried to check the data using "Alluxio fs ls /" and web interface, it shows the data same way in all nodes. I just want to know, is there any way to know (or see) which data file belongs to which host?

thanks again
Mahbub

On Monday, June 25, 2018 at 5:30:52 PM UTC-3, Andrew Audibert wrote:
Hi Mahbub,

Unfortunately there isn't currently a way to specify the host through configuration properties, so SpecificHostPolicy is only usable through the java API where you can instantiate a new SpecificHostPolicy(hostname) and pass it as a method option. You can do that with

FileSystem fs = FileSystem.Factory.get();
OutputStream outStream = fs.createFile(new AlluxioURI("test.csv"), CreateFileOptions.defaults().setLocationPolicy(new SpecificHostPolicy("specific_hostname")));
// write data
outStream.close();

- Andrew

On Mon, Jun 25, 2018 at 7:54 AM Md Mahbub Alam <[hidden email]> wrote:
Hello,

does anyone have any experience in using SpecificHostPolicy(alluxio.client.file.policy.SpecificHostPolicy) to write data into Alluxio in Multi-Node cluster? Currently, I am using Round-Robin Policy. I need to write a specific file (test.csv) to a specific host. I configured the Alluxio with SpecificHostPolicy but how do I select specific host each time I run the program or copy the data into Alluxio. I will appreciate any kind of suggestion.

thanks
Mahbub

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="v4dZss2YBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.