Loading file from HDFS

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading file from HDFS

Fan LIU
Hi,
I am just starting to evaluate Alluxio, which looks very promising to me. I used Alluxio 1.2 built manually against hadoop 2.7.2, copied the core-site.xml from hadoop directory into alluxio/conf, and set underref_address to hdfs://mycluster. When I try to load file from hdfs using: alluxio fs load /path-in-hdfs, I got an error:

java.io.IOException: Path /path-in-hdfs does not exist

    at alluxio.shell.command.WithWildCardPathCommand.run(WithWildCardPathCommand.java:78)

    at alluxio.cli.AlluxioShell.run(AlluxioShell.java:177)

    at alluxio.cli.AlluxioShell.main(AlluxioShell.java:65)


Is the 'load' command suppose to handle my requirement? How can I find more hint about the error?


Thanks,

Fan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Loading file from HDFS

Fan LIU
BTW, I tried loading hdfs from non-cluster hadoop, and it ok. But in HA cluster mode, no luck yet.

On Wednesday, August 3, 2016 at 8:26:39 AM UTC+8, Fan LIU wrote:
Hi,
I am just starting to evaluate Alluxio, which looks very promising to me. I used Alluxio 1.2 built manually against hadoop 2.7.2, copied the core-site.xml from hadoop directory into alluxio/conf, and set underref_address to hdfs://mycluster. When I try to load file from hdfs using: alluxio fs load /path-in-hdfs, I got an error:

java.io.IOException: Path /path-in-hdfs does not exist

    at alluxio.shell.command.WithWildCardPathCommand.run(WithWildCardPathCommand.java:78)

    at alluxio.cli.AlluxioShell.run(AlluxioShell.java:177)

    at alluxio.cli.AlluxioShell.main(AlluxioShell.java:65)


Is the 'load' command suppose to handle my requirement? How can I find more hint about the error?


Thanks,

Fan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Loading file from HDFS

Jiří Šimša
Hello Fan,

The load command can be used to load data from a file stored in UFS (in your case HDFS). If your root UFS is hdfs://hostname:port/<suffix> and the file you wish to load is hdfs://hostname:port/<suffix>/path/to/file, then the command for loading the file should be:

${ALLUXIO_HOME}/bin/alluxio fs load /path/to/file

I have tested that the above works for Alluxio 1.2 and HDFS 2.7.2 running locally on my machine.

Best,

On Tue, Aug 2, 2016 at 11:23 PM, Fan LIU <[hidden email]> wrote:
BTW, I tried loading hdfs from non-cluster hadoop, and it ok. But in HA cluster mode, no luck yet.


On Wednesday, August 3, 2016 at 8:26:39 AM UTC+8, Fan LIU wrote:
Hi,
I am just starting to evaluate Alluxio, which looks very promising to me. I used Alluxio 1.2 built manually against hadoop 2.7.2, copied the core-site.xml from hadoop directory into alluxio/conf, and set underref_address to hdfs://mycluster. When I try to load file from hdfs using: alluxio fs load /path-in-hdfs, I got an error:

java.io.IOException: Path /path-in-hdfs does not exist

    at alluxio.shell.command.WithWildCardPathCommand.run(WithWildCardPathCommand.java:78)

    at alluxio.cli.AlluxioShell.run(AlluxioShell.java:177)

    at alluxio.cli.AlluxioShell.main(AlluxioShell.java:65)


Is the 'load' command suppose to handle my requirement? How can I find more hint about the error?


Thanks,

Fan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Jiří Šimša

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Loading file from HDFS

Fan LIU
Thank you Jiří for the reply.
I've solved the problem. I should copy the hdfs-site.xml into alluxio/conf  from hadoop directory rather than core-site.xml.

I will test the performance of spark on alluxio next. Hope it can surprise me.

Cheers,
Fan  

On Thursday, August 4, 2016 at 11:33:38 PM UTC+8, Jiří Šimša wrote:
Hello Fan,

The load command can be used to load data from a file stored in UFS (in your case HDFS). If your root UFS is hdfs://hostname:port/<suffix> and the file you wish to load is hdfs://hostname:port/<suffix>/path/to/file, then the command for loading the file should be:

${ALLUXIO_HOME}/bin/alluxio fs load /path/to/file

I have tested that the above works for Alluxio 1.2 and HDFS 2.7.2 running locally on my machine.

Best,

On Tue, Aug 2, 2016 at 11:23 PM, Fan LIU <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="wNGBjkdDBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sharp...@...> wrote:
BTW, I tried loading hdfs from non-cluster hadoop, and it ok. But in HA cluster mode, no luck yet.


On Wednesday, August 3, 2016 at 8:26:39 AM UTC+8, Fan LIU wrote:
Hi,
I am just starting to evaluate Alluxio, which looks very promising to me. I used Alluxio 1.2 built manually against hadoop 2.7.2, copied the core-site.xml from hadoop directory into alluxio/conf, and set underref_address to hdfs://mycluster. When I try to load file from hdfs using: alluxio fs load /path-in-hdfs, I got an error:

java.io.IOException: Path /path-in-hdfs does not exist

    at alluxio.shell.command.WithWildCardPathCommand.run(WithWildCardPathCommand.java:78)

    at alluxio.cli.AlluxioShell.run(AlluxioShell.java:177)

    at alluxio.cli.AlluxioShell.main(AlluxioShell.java:65)


Is the 'load' command suppose to handle my requirement? How can I find more hint about the error?


Thanks,

Fan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="wNGBjkdDBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Jiří Šimša
<a href="http://alluxio.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBX05zcSsMVKyYu5VW-WXWhLx2Sg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBX05zcSsMVKyYu5VW-WXWhLx2Sg&#39;;return true;">Alluxio, Inc.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.