Can't get GCS integration to work?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't get GCS integration to work?

Paul Wais
Dear List,


I'm trying to mount my Google Storage bucket using a local instance of Alluxio. I can mount the directory just fine:


ALLUXIO_MASTER_HOSTNAME=192.168.99.100 /opt/alluxio/bin/alluxio fs mount --option fs.gcs.accessKeyId=GOOG.... --option fs.gcs.secretAccessKey=rGV..... /gcs gs://my/bucket


But when I try to view the files in the web browser or fuse-mount alluxio, I see nothing.  I do see this in fuse.log:

2018-10-08 05:35:13,461 ERROR AlluxioFuseUtils - Failed to get id from unknown with option -u
2018-10-08 05:35:13,464 ERROR AlluxioFuseUtils - Failed to get id from unknown with option -g
2018-10-08 05:35:17,300 ERROR AlluxioFuseUtils - Failed to get id from unknown with option -u


I believe my gcs bucket has the right permissions... "Owners of project" has Storage Object {Admin,Creator,Viewer} permissions.  The bucket is not public.  

Anybody have any ideas?  I'm just learning to troubleshoot gcloud stuff like this.

Cheers,
-Paul




--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Pardon,

I'm on alluxio 1.8.1, java 1.8, ubuntu 16.04.   This is in docker on a Mac.  


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Bin Fan
hi Paul

after mount, are you able to run cmd like this:
/opt/alluxio/bin/alluxio runTests --directory /gcs 

if no, then the mount is not working. check it out by:
/opt/alluxio/bin/alluxio fs mount

if yes, there might be something wrong with your fuse integration.
can you show your cmd to run fuse?

On Sat, Oct 13, 2018 at 1:30 AM Paul Wais <[hidden email]> wrote:
Pardon,

I'm on alluxio 1.8.1, java 1.8, ubuntu 16.04.   This is in docker on a Mac.  


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
We have already begun to address the uid/gid issue and hopefully, it will be solved soon.

Thanks,
Lu


On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Lu,

Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?

Cheers,
-Paul

On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
Hi Paul,

The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
We have already begun to address the uid/gid issue and hopefully, it will be solved soon.

Thanks,
Lu


On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

Two properties mainly affect Alluxio caching:
1) alluxio.user.file.writetype.default  which is MUST_CACHE by default.
Default write type when creating Alluxio files. Valid options are `MUST_CACHE` (write will only go to Alluxio and must be stored in Alluxio), `CACHE_THROUGH` (try to cache, write to UnderFS synchronously), `THROUGH` (no cache, write to UnderFS synchronously).

2) alluxio.user.file.readtype.default which is CACHE_PROMOTE by default
Default read type when creating Alluxio files. Valid options are `CACHE_PROMOTE` (move data to highest tier if already in Alluxio storage, write data into highest tier of local Alluxio if data needs to be read from under storage), `CACHE` (write data into highest tier of local Alluxio if data needs to be read from under storage), `NO_CACHE` (no data interaction with Alluxio, if the read is from Alluxio data migration or eviction will not occur).

If you want to change their values, you could set them in the ${ALLUXIO_HOME}/conf/alluxio-site.properties.

In your case, you already make use of the Alluxio caching!

Thanks,
Lu

On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?

Cheers,
-Paul

On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
Hi Paul,

The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
We have already begun to address the uid/gid issue and hopefully, it will be solved soon.

Thanks,
Lu


On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
In reply to this post by Paul Wais
Sorry Paul, I misunderstood your question. I will try to reproduce your problem  
and see if there are any issue with Alluxio Fuse caching.

Thanks,
Lu

On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?

Cheers,
-Paul

On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
Hi Paul,

The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
We have already begun to address the uid/gid issue and hopefully, it will be solved soon.

Thanks,
Lu


On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

Could you share your logs for us to figure out the problem?
The master.log, worker.log, fuse.log, and maybe other logs if possible.

Thanks,
Lu

On Mon, Oct 15, 2018 at 2:09 PM Lu <[hidden email]> wrote:
Sorry Paul, I misunderstood your question. I will try to reproduce your problem  
and see if there are any issue with Alluxio Fuse caching.

Thanks,
Lu

On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?

Cheers,
-Paul

On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
Hi Paul,

The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
We have already begun to address the uid/gid issue and hopefully, it will be solved soon.

Thanks,
Lu


On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(

Cheers,
-Paul

On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
Hi Paul,

Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.

It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed. 

I created a jira to track this issue and will work to figure out what's the best approach to solving it.

Thanks for reporting this issue!
Lu



On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
Hi Bin,


Thanks for the super fast response!

That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)


I have a new issue though: I do not appear to see alluxio fuse caching files:

root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
28950 /mnt/alluxio /


root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m20.418s
user 0m0.000s
sys 0m0.160s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m9.134s
user 0m0.000s
sys 0m0.430s

root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
real 0m21.445s
user 0m0.000s
sys 0m0.180s


root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
real 0m0.062s
user 0m0.000s
sys 0m0.060s

root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip

In the alluxio webui, I see the files but the "in alluxio" percentage is 0%

I had expected alluxio to use local disk caching.   Here's my config:



alluxio.underfs.address=/opt/alluxio-underfs

// Define 2 storage levels.
alluxio.worker.tieredstore.levels=2

// Define MEM as the lowest level.
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
alluxio.worker.tieredstore.level0.dirs.quota=64M
alluxio.worker.tieredstore.level0.reserved.ratio=0.1

// Define SSD as the secondary level.
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/tmp
alluxio.worker.tieredstore.level1.dirs.quota=2TB
alluxio.worker.tieredstore.level1.reserved.ratio=0.1



My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html  

Any ideas?  clearly I must have configured the cache wrong somehow.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Lu,

Thanks for all these details!  Are these settings mentioned in the
docs somewhere?  (Or mebbe may I create a PR to add them if not?)

I definitely have alluxio.user.file.readtype.default=CACHE_PROMOTE

Here are my logs: https://gist.github.com/pwais/98bc1fc18a450ff407fb22c03ce3912d

It might be easier for me to debug this locally.  Can you confirm that
a simple 'cat' is an OK way to attempt to trigger caching?  If so then
maybe I can try to enable debug logging and poke from there.

Cheers,
-Paul

On Mon, Oct 15, 2018 at 2:13 PM Lu <[hidden email]> wrote:

>
> Hi Paul,
>
> Could you share your logs for us to figure out the problem?
> The master.log, worker.log, fuse.log, and maybe other logs if possible.
>
> Thanks,
> Lu
>
> On Mon, Oct 15, 2018 at 2:09 PM Lu <[hidden email]> wrote:
>>
>> Sorry Paul, I misunderstood your question. I will try to reproduce your problem
>> and see if there are any issue with Alluxio Fuse caching.
>>
>> Thanks,
>> Lu
>>
>> On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
>>>
>>> Hi Lu,
>>>
>>> Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?
>>>
>>> Cheers,
>>> -Paul
>>>
>>> On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
>>>> We have already begun to address the uid/gid issue and hopefully, it will be solved soon.
>>>>
>>>> Thanks,
>>>> Lu
>>>>
>>>>
>>>> On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
>>>>>
>>>>> Hi Lu,
>>>>>
>>>>> Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(
>>>>>
>>>>> Cheers,
>>>>> -Paul
>>>>>
>>>>> On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
>>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
>>>>>> In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.
>>>>>>
>>>>>> It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
>>>>>> log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed.
>>>>>>
>>>>>> I created a jira to track this issue and will work to figure out what's the best approach to solving it.
>>>>>>
>>>>>> Thanks for reporting this issue!
>>>>>> Lu
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi Bin,
>>>>>>>
>>>>>>>
>>>>>>> Thanks for the super fast response!
>>>>>>>
>>>>>>> That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)
>>>>>>>
>>>>>>>
>>>>>>> I have a new issue though: I do not appear to see alluxio fuse caching files:
>>>>>>>
>>>>>>> root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
>>>>>>> pid mount_point alluxio_path
>>>>>>> 28950 /mnt/alluxio /
>>>>>>>
>>>>>>>
>>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
>>>>>>> real 0m20.418s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.160s
>>>>>>>
>>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
>>>>>>> real 0m9.134s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.430s
>>>>>>>
>>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
>>>>>>> real 0m21.445s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.180s
>>>>>>>
>>>>>>>
>>>>>>> root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
>>>>>>> real 0m0.062s
>>>>>>> user 0m0.000s
>>>>>>> sys 0m0.060s
>>>>>>>
>>>>>>> root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
>>>>>>> ---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip
>>>>>>>
>>>>>>> In the alluxio webui, I see the files but the "in alluxio" percentage is 0%
>>>>>>>
>>>>>>> I had expected alluxio to use local disk caching.   Here's my config:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> alluxio.underfs.address=/opt/alluxio-underfs
>>>>>>>
>>>>>>> // Define 2 storage levels.
>>>>>>> alluxio.worker.tieredstore.levels=2
>>>>>>>
>>>>>>> // Define MEM as the lowest level.
>>>>>>> alluxio.worker.tieredstore.level0.alias=MEM
>>>>>>> alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
>>>>>>> alluxio.worker.tieredstore.level0.dirs.quota=64M
>>>>>>> alluxio.worker.tieredstore.level0.reserved.ratio=0.1
>>>>>>>
>>>>>>> // Define SSD as the secondary level.
>>>>>>> alluxio.worker.tieredstore.level1.alias=SSD
>>>>>>> alluxio.worker.tieredstore.level1.dirs.path=/tmp
>>>>>>> alluxio.worker.tieredstore.level1.dirs.quota=2TB
>>>>>>> alluxio.worker.tieredstore.level1.reserved.ratio=0.1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html
>>>>>>>
>>>>>>> Any ideas?  clearly I must have configured the cache wrong somehow.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>>>>>>>
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
FWIW, I do see /gcs/default_tests_files/BASIC_NO_CACHE_CACHE_THROUGH
is 100% in Alluxio, so it was able to cache stuff the tests wrote.
On Mon, Oct 15, 2018 at 2:53 PM Paul Wais <[hidden email]> wrote:

>
> Hi Lu,
>
> Thanks for all these details!  Are these settings mentioned in the
> docs somewhere?  (Or mebbe may I create a PR to add them if not?)
>
> I definitely have alluxio.user.file.readtype.default=CACHE_PROMOTE
>
> Here are my logs: https://gist.github.com/pwais/98bc1fc18a450ff407fb22c03ce3912d
>
> It might be easier for me to debug this locally.  Can you confirm that
> a simple 'cat' is an OK way to attempt to trigger caching?  If so then
> maybe I can try to enable debug logging and poke from there.
>
> Cheers,
> -Paul
>
> On Mon, Oct 15, 2018 at 2:13 PM Lu <[hidden email]> wrote:
> >
> > Hi Paul,
> >
> > Could you share your logs for us to figure out the problem?
> > The master.log, worker.log, fuse.log, and maybe other logs if possible.
> >
> > Thanks,
> > Lu
> >
> > On Mon, Oct 15, 2018 at 2:09 PM Lu <[hidden email]> wrote:
> >>
> >> Sorry Paul, I misunderstood your question. I will try to reproduce your problem
> >> and see if there are any issue with Alluxio Fuse caching.
> >>
> >> Thanks,
> >> Lu
> >>
> >> On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
> >>>
> >>> Hi Lu,
> >>>
> >>> Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?
> >>>
> >>> Cheers,
> >>> -Paul
> >>>
> >>> On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
> >>>>
> >>>> Hi Paul,
> >>>>
> >>>> The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
> >>>> We have already begun to address the uid/gid issue and hopefully, it will be solved soon.
> >>>>
> >>>> Thanks,
> >>>> Lu
> >>>>
> >>>>
> >>>> On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
> >>>>>
> >>>>> Hi Lu,
> >>>>>
> >>>>> Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(
> >>>>>
> >>>>> Cheers,
> >>>>> -Paul
> >>>>>
> >>>>> On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
> >>>>>>
> >>>>>> Hi Paul,
> >>>>>>
> >>>>>> Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
> >>>>>> In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.
> >>>>>>
> >>>>>> It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
> >>>>>> log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed.
> >>>>>>
> >>>>>> I created a jira to track this issue and will work to figure out what's the best approach to solving it.
> >>>>>>
> >>>>>> Thanks for reporting this issue!
> >>>>>> Lu
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> Hi Bin,
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks for the super fast response!
> >>>>>>>
> >>>>>>> That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)
> >>>>>>>
> >>>>>>>
> >>>>>>> I have a new issue though: I do not appear to see alluxio fuse caching files:
> >>>>>>>
> >>>>>>> root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
> >>>>>>> pid mount_point alluxio_path
> >>>>>>> 28950 /mnt/alluxio /
> >>>>>>>
> >>>>>>>
> >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> >>>>>>> real 0m20.418s
> >>>>>>> user 0m0.000s
> >>>>>>> sys 0m0.160s
> >>>>>>>
> >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> >>>>>>> real 0m9.134s
> >>>>>>> user 0m0.000s
> >>>>>>> sys 0m0.430s
> >>>>>>>
> >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> >>>>>>> real 0m21.445s
> >>>>>>> user 0m0.000s
> >>>>>>> sys 0m0.180s
> >>>>>>>
> >>>>>>>
> >>>>>>> root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
> >>>>>>> real 0m0.062s
> >>>>>>> user 0m0.000s
> >>>>>>> sys 0m0.060s
> >>>>>>>
> >>>>>>> root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
> >>>>>>> ---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip
> >>>>>>>
> >>>>>>> In the alluxio webui, I see the files but the "in alluxio" percentage is 0%
> >>>>>>>
> >>>>>>> I had expected alluxio to use local disk caching.   Here's my config:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> alluxio.underfs.address=/opt/alluxio-underfs
> >>>>>>>
> >>>>>>> // Define 2 storage levels.
> >>>>>>> alluxio.worker.tieredstore.levels=2
> >>>>>>>
> >>>>>>> // Define MEM as the lowest level.
> >>>>>>> alluxio.worker.tieredstore.level0.alias=MEM
> >>>>>>> alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
> >>>>>>> alluxio.worker.tieredstore.level0.dirs.quota=64M
> >>>>>>> alluxio.worker.tieredstore.level0.reserved.ratio=0.1
> >>>>>>>
> >>>>>>> // Define SSD as the secondary level.
> >>>>>>> alluxio.worker.tieredstore.level1.alias=SSD
> >>>>>>> alluxio.worker.tieredstore.level1.dirs.path=/tmp
> >>>>>>> alluxio.worker.tieredstore.level1.dirs.quota=2TB
> >>>>>>> alluxio.worker.tieredstore.level1.reserved.ratio=0.1
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html
> >>>>>>>
> >>>>>>> Any ideas?  clearly I must have configured the cache wrong somehow.
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> >>>>>>>
> >>>>>>>
> >>>>>>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Lu,

I just added a small 512kb text file and I was able to use `cat` to
get the data to be "in alluxio".

Does CACHE_PROMOTE mean the file would have to get promoted to RAM? I
don't have a very large RAM tier, but I have a large SSD tier.  Maybe
that could be the problem?  Ideally I'd want the Google-Cloud-read
data to at least be cached in the SSD tier.

So my hypothesis is that if a file is too big for the memory tier,
then it'll get dropped entirely instead of made a candidate for any
lower tiers.  I'm hoping I'm wrong, but can you confirm?

I'm going to try turning off the memory tier and I'll see what
happens.  FWIW if I use fuse, I'd think the OS disk cache would work
fine for me versus having an explicit memory tier in Alluxio.

Cheers,
-Paul

On Mon, Oct 15, 2018 at 2:54 PM Paul Wais <[hidden email]> wrote:

>
> FWIW, I do see /gcs/default_tests_files/BASIC_NO_CACHE_CACHE_THROUGH
> is 100% in Alluxio, so it was able to cache stuff the tests wrote.
> On Mon, Oct 15, 2018 at 2:53 PM Paul Wais <[hidden email]> wrote:
> >
> > Hi Lu,
> >
> > Thanks for all these details!  Are these settings mentioned in the
> > docs somewhere?  (Or mebbe may I create a PR to add them if not?)
> >
> > I definitely have alluxio.user.file.readtype.default=CACHE_PROMOTE
> >
> > Here are my logs: https://gist.github.com/pwais/98bc1fc18a450ff407fb22c03ce3912d
> >
> > It might be easier for me to debug this locally.  Can you confirm that
> > a simple 'cat' is an OK way to attempt to trigger caching?  If so then
> > maybe I can try to enable debug logging and poke from there.
> >
> > Cheers,
> > -Paul
> >
> > On Mon, Oct 15, 2018 at 2:13 PM Lu <[hidden email]> wrote:
> > >
> > > Hi Paul,
> > >
> > > Could you share your logs for us to figure out the problem?
> > > The master.log, worker.log, fuse.log, and maybe other logs if possible.
> > >
> > > Thanks,
> > > Lu
> > >
> > > On Mon, Oct 15, 2018 at 2:09 PM Lu <[hidden email]> wrote:
> > >>
> > >> Sorry Paul, I misunderstood your question. I will try to reproduce your problem
> > >> and see if there are any issue with Alluxio Fuse caching.
> > >>
> > >> Thanks,
> > >> Lu
> > >>
> > >> On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
> > >>>
> > >>> Hi Lu,
> > >>>
> > >>> Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?
> > >>>
> > >>> Cheers,
> > >>> -Paul
> > >>>
> > >>> On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
> > >>>>
> > >>>> Hi Paul,
> > >>>>
> > >>>> The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
> > >>>> We have already begun to address the uid/gid issue and hopefully, it will be solved soon.
> > >>>>
> > >>>> Thanks,
> > >>>> Lu
> > >>>>
> > >>>>
> > >>>> On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
> > >>>>>
> > >>>>> Hi Lu,
> > >>>>>
> > >>>>> Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(
> > >>>>>
> > >>>>> Cheers,
> > >>>>> -Paul
> > >>>>>
> > >>>>> On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
> > >>>>>>
> > >>>>>> Hi Paul,
> > >>>>>>
> > >>>>>> Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
> > >>>>>> In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.
> > >>>>>>
> > >>>>>> It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
> > >>>>>> log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed.
> > >>>>>>
> > >>>>>> I created a jira to track this issue and will work to figure out what's the best approach to solving it.
> > >>>>>>
> > >>>>>> Thanks for reporting this issue!
> > >>>>>> Lu
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
> > >>>>>>>
> > >>>>>>> Hi Bin,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Thanks for the super fast response!
> > >>>>>>>
> > >>>>>>> That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I have a new issue though: I do not appear to see alluxio fuse caching files:
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
> > >>>>>>> pid mount_point alluxio_path
> > >>>>>>> 28950 /mnt/alluxio /
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > >>>>>>> real 0m20.418s
> > >>>>>>> user 0m0.000s
> > >>>>>>> sys 0m0.160s
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > >>>>>>> real 0m9.134s
> > >>>>>>> user 0m0.000s
> > >>>>>>> sys 0m0.430s
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > >>>>>>> real 0m21.445s
> > >>>>>>> user 0m0.000s
> > >>>>>>> sys 0m0.180s
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
> > >>>>>>> real 0m0.062s
> > >>>>>>> user 0m0.000s
> > >>>>>>> sys 0m0.060s
> > >>>>>>>
> > >>>>>>> root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
> > >>>>>>> ---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip
> > >>>>>>>
> > >>>>>>> In the alluxio webui, I see the files but the "in alluxio" percentage is 0%
> > >>>>>>>
> > >>>>>>> I had expected alluxio to use local disk caching.   Here's my config:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> alluxio.underfs.address=/opt/alluxio-underfs
> > >>>>>>>
> > >>>>>>> // Define 2 storage levels.
> > >>>>>>> alluxio.worker.tieredstore.levels=2
> > >>>>>>>
> > >>>>>>> // Define MEM as the lowest level.
> > >>>>>>> alluxio.worker.tieredstore.level0.alias=MEM
> > >>>>>>> alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
> > >>>>>>> alluxio.worker.tieredstore.level0.dirs.quota=64M
> > >>>>>>> alluxio.worker.tieredstore.level0.reserved.ratio=0.1
> > >>>>>>>
> > >>>>>>> // Define SSD as the secondary level.
> > >>>>>>> alluxio.worker.tieredstore.level1.alias=SSD
> > >>>>>>> alluxio.worker.tieredstore.level1.dirs.path=/tmp
> > >>>>>>> alluxio.worker.tieredstore.level1.dirs.quota=2TB
> > >>>>>>> alluxio.worker.tieredstore.level1.reserved.ratio=0.1
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html
> > >>>>>>>
> > >>>>>>> Any ideas?  clearly I must have configured the cache wrong somehow.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>>
> > >>>>>>> You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
> > >>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Derp, the worker did not start for me when I simply removed the MEM tier ...
On Mon, Oct 15, 2018 at 3:06 PM Paul Wais <[hidden email]> wrote:

>
> Hi Lu,
>
> I just added a small 512kb text file and I was able to use `cat` to
> get the data to be "in alluxio".
>
> Does CACHE_PROMOTE mean the file would have to get promoted to RAM? I
> don't have a very large RAM tier, but I have a large SSD tier.  Maybe
> that could be the problem?  Ideally I'd want the Google-Cloud-read
> data to at least be cached in the SSD tier.
>
> So my hypothesis is that if a file is too big for the memory tier,
> then it'll get dropped entirely instead of made a candidate for any
> lower tiers.  I'm hoping I'm wrong, but can you confirm?
>
> I'm going to try turning off the memory tier and I'll see what
> happens.  FWIW if I use fuse, I'd think the OS disk cache would work
> fine for me versus having an explicit memory tier in Alluxio.
>
> Cheers,
> -Paul
>
> On Mon, Oct 15, 2018 at 2:54 PM Paul Wais <[hidden email]> wrote:
> >
> > FWIW, I do see /gcs/default_tests_files/BASIC_NO_CACHE_CACHE_THROUGH
> > is 100% in Alluxio, so it was able to cache stuff the tests wrote.
> > On Mon, Oct 15, 2018 at 2:53 PM Paul Wais <[hidden email]> wrote:
> > >
> > > Hi Lu,
> > >
> > > Thanks for all these details!  Are these settings mentioned in the
> > > docs somewhere?  (Or mebbe may I create a PR to add them if not?)
> > >
> > > I definitely have alluxio.user.file.readtype.default=CACHE_PROMOTE
> > >
> > > Here are my logs: https://gist.github.com/pwais/98bc1fc18a450ff407fb22c03ce3912d
> > >
> > > It might be easier for me to debug this locally.  Can you confirm that
> > > a simple 'cat' is an OK way to attempt to trigger caching?  If so then
> > > maybe I can try to enable debug logging and poke from there.
> > >
> > > Cheers,
> > > -Paul
> > >
> > > On Mon, Oct 15, 2018 at 2:13 PM Lu <[hidden email]> wrote:
> > > >
> > > > Hi Paul,
> > > >
> > > > Could you share your logs for us to figure out the problem?
> > > > The master.log, worker.log, fuse.log, and maybe other logs if possible.
> > > >
> > > > Thanks,
> > > > Lu
> > > >
> > > > On Mon, Oct 15, 2018 at 2:09 PM Lu <[hidden email]> wrote:
> > > >>
> > > >> Sorry Paul, I misunderstood your question. I will try to reproduce your problem
> > > >> and see if there are any issue with Alluxio Fuse caching.
> > > >>
> > > >> Thanks,
> > > >> Lu
> > > >>
> > > >> On Mon, Oct 15, 2018 at 1:37 PM Paul Wais <[hidden email]> wrote:
> > > >>>
> > > >>> Hi Lu,
> > > >>>
> > > >>> Based upon the shell session I pasted, it does not look like Alluxio is caching at all.  Am I missing something?
> > > >>>
> > > >>> Cheers,
> > > >>> -Paul
> > > >>>
> > > >>> On Mon, Oct 15, 2018 at 13:32 Lu <[hidden email]> wrote:
> > > >>>>
> > > >>>> Hi Paul,
> > > >>>>
> > > >>>> The Fuse uid/gid only affect the `ls/chown/chgrp` shell commands. Alluxio caching works well!
> > > >>>> We have already begun to address the uid/gid issue and hopefully, it will be solved soon.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Lu
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Oct 15, 2018 at 1:28 PM Paul Wais <[hidden email]> wrote:
> > > >>>>>
> > > >>>>> Hi Lu,
> > > >>>>>
> > > >>>>> Thanksgir the info!  What about the caching though?  I can’t use Alluxio if it won’t cache :(
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> -Paul
> > > >>>>>
> > > >>>>> On Mon, Oct 15, 2018 at 10:16 Lu <[hidden email]> wrote:
> > > >>>>>>
> > > >>>>>> Hi Paul,
> > > >>>>>>
> > > >>>>>> Alluxio Fuse will try to find user and group from the underlying filesystem(the mounted path).
> > > >>>>>> In your case, it seems that Alluxio Fuse failed in matching GCS users/groups with Alluxio.
> > > >>>>>>
> > > >>>>>> It doesn't affect Alluxio Fuse normal read/write functionalities. You can disable the log message by adding
> > > >>>>>> log4j.logger.alluxio.fuse.AlluxioFuseUtils=OFF to conf/log4j.properties under the alluxio home folder. After updating the property, you'll need to re-mount fuse for it to pick up the change. The lookups will still be attempted, but at least your log won't be spammed.
> > > >>>>>>
> > > >>>>>> I created a jira to track this issue and will work to figure out what's the best approach to solving it.
> > > >>>>>>
> > > >>>>>> Thanks for reporting this issue!
> > > >>>>>> Lu
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Sat, Oct 13, 2018 at 3:09 PM Paul Wais <[hidden email]> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Bin,
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Thanks for the super fast response!
> > > >>>>>>>
> > > >>>>>>> That runTests command worked (thanks for showing me how to use a target directory!!), and that helped me figure out that I had put my data in the wrong bucket :( :( :(  So now I can indeed read the data :)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> I have a new issue though: I do not appear to see alluxio fuse caching files:
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# /opt/alluxio/integration/fuse/bin/alluxio-fuse stat
> > > >>>>>>> pid mount_point alluxio_path
> > > >>>>>>> 28950 /mnt/alluxio /
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > > >>>>>>> real 0m20.418s
> > > >>>>>>> user 0m0.000s
> > > >>>>>>> sys 0m0.160s
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > > >>>>>>> real 0m9.134s
> > > >>>>>>> user 0m0.000s
> > > >>>>>>> sys 0m0.430s
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# time cat /mnt/alluxio/gcs/my_file.zip > /tmp/yay
> > > >>>>>>> real 0m21.445s
> > > >>>>>>> user 0m0.000s
> > > >>>>>>> sys 0m0.180s
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# time cat /tmp/yay > /tmp/yay2
> > > >>>>>>> real 0m0.062s
> > > >>>>>>> user 0m0.000s
> > > >>>>>>> sys 0m0.060s
> > > >>>>>>>
> > > >>>>>>> root@default:/opt/au# ls -lhat /mnt/alluxio/gcs/my_file.zip
> > > >>>>>>> ---------- 0 nobody nogroup 107M Oct 13 21:30 /mnt/alluxio/gcs/my_file.zip
> > > >>>>>>>
> > > >>>>>>> In the alluxio webui, I see the files but the "in alluxio" percentage is 0%
> > > >>>>>>>
> > > >>>>>>> I had expected alluxio to use local disk caching.   Here's my config:
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> alluxio.underfs.address=/opt/alluxio-underfs
> > > >>>>>>>
> > > >>>>>>> // Define 2 storage levels.
> > > >>>>>>> alluxio.worker.tieredstore.levels=2
> > > >>>>>>>
> > > >>>>>>> // Define MEM as the lowest level.
> > > >>>>>>> alluxio.worker.tieredstore.level0.alias=MEM
> > > >>>>>>> alluxio.worker.tieredstore.level0.dirs.path=/dev/shm
> > > >>>>>>> alluxio.worker.tieredstore.level0.dirs.quota=64M
> > > >>>>>>> alluxio.worker.tieredstore.level0.reserved.ratio=0.1
> > > >>>>>>>
> > > >>>>>>> // Define SSD as the secondary level.
> > > >>>>>>> alluxio.worker.tieredstore.level1.alias=SSD
> > > >>>>>>> alluxio.worker.tieredstore.level1.dirs.path=/tmp
> > > >>>>>>> alluxio.worker.tieredstore.level1.dirs.quota=2TB
> > > >>>>>>> alluxio.worker.tieredstore.level1.reserved.ratio=0.1
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> My use case is similar to your tensorflow use case listed here: https://www.alluxio.org/docs/1.8/en/Running-Deep-Learning-on-Alluxio.html
> > > >>>>>>>
> > > >>>>>>> Any ideas?  clearly I must have configured the cache wrong somehow.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>>
> > > >>>>>>> You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
> > > >>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

Alluxio cache files to the first tier and then eviction may happen to put files to the other tiers(like in your case, the SSD tier). If the file size is bigger than the size of the first tier, we are not able to cache this file.

If you want to cache your file in the SSD, you could set the SSD as the first tier.

alluxio.worker.tieredstore.levels=1

alluxio.worker.tieredstore.level0.alias=SSD
alluxio.worker.tieredstore.level0.dirs.path=<your_path>
alluxio.worker.tieredstore.level0.dirs.quota=<your_ssd_size>
alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9 
alluxio.worker.tieredstore.level0.watermark.low.ratio=0.75

Note that the reserved.ratio has been deprecated long ago, and the new configuration is watermark.high.ratio and watermark.low.ratio.

If it still not work, please share us your worker logs and configuration settings(alluxio-site.properties).

Thanks,
Lu

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Paul Wais
Hi Lu,

I got it working! wow!! Thank you so much!! This is a giant help
because google's own FUSE client doesn't do read caching at all ( :P
), and s3fs appears to be broken for GCS (based upon my own tests) :(
I wanted Alluxio anyways, but the Alluxio FUSE client is a major bonus
and helps me develop my app much much faster.

I'm curious:  if a user has a SSD / disk tier, is a memory tier
necessary?  I would think the OS disk cache would serve as an
effective and automatic cache layer for the disk tier (even for
multiple worker processes).  I can understand there might be some
applications where the user wants to pin in memory thru Alluxio, or
there's a ramdisk they want to persist between Alluxio runs (or
Alluxio container lifetimes).

One potential bug:
FWIW when I define level1 before level0 (i.e uncomment the commented
section in the following gist), I get a NullPointerException in the
worker: https://gist.github.com/pwais/5c6ac996f12dbdd53e9a7f564237b4c8

When I comment the section, I get no NPE.


Also I wanted to recommend you might look at the kubespray playbook
for gluster: https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/network-storage/glusterfs
    I think having an Alluxio playbook along side the gluster option
would be pretty amazing.  I've used PB-scale Gluster and it's not too
bad, and I've used gluster on NVME and the performance is great.  I
think a very cost-effective and performant EC2 deployment would be
gluster+alluxio across a bunch of i3 instances.  (I conjecture that
might be similar to what Databricks does for dbfs:// under the hood ;)

Cheers,
-Paul



On Mon, Oct 15, 2018 at 4:28 PM Lu <[hidden email]> wrote:

>
> Hi Paul,
>
> Alluxio cache files to the first tier and then eviction may happen to put files to the other tiers(like in your case, the SSD tier). If the file size is bigger than the size of the first tier, we are not able to cache this file.
>
> If you want to cache your file in the SSD, you could set the SSD as the first tier.
>
> alluxio.worker.tieredstore.levels=1
>
> alluxio.worker.tieredstore.level0.alias=SSD
> alluxio.worker.tieredstore.level0.dirs.path=<your_path>
> alluxio.worker.tieredstore.level0.dirs.quota=<your_ssd_size>
> alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9
> alluxio.worker.tieredstore.level0.watermark.low.ratio=0.75
>
> Note that the reserved.ratio has been deprecated long ago, and the new configuration is watermark.high.ratio and watermark.low.ratio.
>
> If it still not work, please share us your worker logs and configuration settings(alluxio-site.properties).
>
> Thanks,
> Lu
>

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Lu Qiu
Hi Paul,

Glad to hear that Alluxio is a great choice for GCS with our read caching ability and Fuse integration!!

Users can configure Alluxio with only SSD/disk tier(no memory tier) and we are experimenting with this kind of use cases. In the current stage, we recommend users to use a memory tier as the first tier because of the cache speed, but memory tier is not a must-have.

The NPE may happen because of the wrong tier number.
alluxio.worker.tieredstore.levels=1
If two tiers are provided, the number should be changed from 1 to 2.

Thanks for your advice! The kubespray playbook for gluster is quite convenient. We would consider those kinds of playbook with different ufs to make it easier for our users.









On Mon, Oct 15, 2018 at 10:48 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

I got it working! wow!! Thank you so much!! This is a giant help
because google's own FUSE client doesn't do read caching at all ( :P
), and s3fs appears to be broken for GCS (based upon my own tests) :(
I wanted Alluxio anyways, but the Alluxio FUSE client is a major bonus
and helps me develop my app much much faster.

I'm curious:  if a user has a SSD / disk tier, is a memory tier
necessary?  I would think the OS disk cache would serve as an
effective and automatic cache layer for the disk tier (even for
multiple worker processes).  I can understand there might be some
applications where the user wants to pin in memory thru Alluxio, or
there's a ramdisk they want to persist between Alluxio runs (or
Alluxio container lifetimes).

One potential bug:
FWIW when I define level1 before level0 (i.e uncomment the commented
section in the following gist), I get a NullPointerException in the
worker: https://gist.github.com/pwais/5c6ac996f12dbdd53e9a7f564237b4c8

When I comment the section, I get no NPE.


Also I wanted to recommend you might look at the kubespray playbook
for gluster: https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/network-storage/glusterfs
    I think having an Alluxio playbook along side the gluster option
would be pretty amazing.  I've used PB-scale Gluster and it's not too
bad, and I've used gluster on NVME and the performance is great.  I
think a very cost-effective and performant EC2 deployment would be
gluster+alluxio across a bunch of i3 instances.  (I conjecture that
might be similar to what Databricks does for dbfs:// under the hood ;)

Cheers,
-Paul



On Mon, Oct 15, 2018 at 4:28 PM Lu <[hidden email]> wrote:
>
> Hi Paul,
>
> Alluxio cache files to the first tier and then eviction may happen to put files to the other tiers(like in your case, the SSD tier). If the file size is bigger than the size of the first tier, we are not able to cache this file.
>
> If you want to cache your file in the SSD, you could set the SSD as the first tier.
>
> alluxio.worker.tieredstore.levels=1
>
> alluxio.worker.tieredstore.level0.alias=SSD
> alluxio.worker.tieredstore.level0.dirs.path=<your_path>
> alluxio.worker.tieredstore.level0.dirs.quota=<your_ssd_size>
> alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9
> alluxio.worker.tieredstore.level0.watermark.low.ratio=0.75
>
> Note that the reserved.ratio has been deprecated long ago, and the new configuration is watermark.high.ratio and watermark.low.ratio.
>
> If it still not work, please share us your worker logs and configuration settings(alluxio-site.properties).
>
> Thanks,
> Lu
>

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't get GCS integration to work?

Andrew Audibert
Hi Paul,

A memory tier can help by giving you more consistent performance and control/visibility around what data is stored in memory. However, if the memory tier is too small relative to the size of your working set, Alluxio will have to evict a lot of data to the SSD tier, potentially creating a performance bottleneck. OS buffer cache + SSD-only Alluxio is easier to manage while often coming close to the performance of having both SSD and MEM tiers. If your memory tier would be small, I recommend choosing a single-tier SSD setup.

Thanks for reporting the NPE from tiered storage configuration! If you run into any more bugs or rough edges, please file a ticket on our JIRA to help us track and fix.

Cheers,
Andrew

On Tue, Oct 16, 2018 at 10:58 AM Lu <[hidden email]> wrote:
Hi Paul,

Glad to hear that Alluxio is a great choice for GCS with our read caching ability and Fuse integration!!

Users can configure Alluxio with only SSD/disk tier(no memory tier) and we are experimenting with this kind of use cases. In the current stage, we recommend users to use a memory tier as the first tier because of the cache speed, but memory tier is not a must-have.

The NPE may happen because of the wrong tier number.
alluxio.worker.tieredstore.levels=1
If two tiers are provided, the number should be changed from 1 to 2.

Thanks for your advice! The kubespray playbook for gluster is quite convenient. We would consider those kinds of playbook with different ufs to make it easier for our users.









On Mon, Oct 15, 2018 at 10:48 PM Paul Wais <[hidden email]> wrote:
Hi Lu,

I got it working! wow!! Thank you so much!! This is a giant help
because google's own FUSE client doesn't do read caching at all ( :P
), and s3fs appears to be broken for GCS (based upon my own tests) :(
I wanted Alluxio anyways, but the Alluxio FUSE client is a major bonus
and helps me develop my app much much faster.

I'm curious:  if a user has a SSD / disk tier, is a memory tier
necessary?  I would think the OS disk cache would serve as an
effective and automatic cache layer for the disk tier (even for
multiple worker processes).  I can understand there might be some
applications where the user wants to pin in memory thru Alluxio, or
there's a ramdisk they want to persist between Alluxio runs (or
Alluxio container lifetimes).

One potential bug:
FWIW when I define level1 before level0 (i.e uncomment the commented
section in the following gist), I get a NullPointerException in the
worker: https://gist.github.com/pwais/5c6ac996f12dbdd53e9a7f564237b4c8

When I comment the section, I get no NPE.


Also I wanted to recommend you might look at the kubespray playbook
for gluster: https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/network-storage/glusterfs
    I think having an Alluxio playbook along side the gluster option
would be pretty amazing.  I've used PB-scale Gluster and it's not too
bad, and I've used gluster on NVME and the performance is great.  I
think a very cost-effective and performant EC2 deployment would be
gluster+alluxio across a bunch of i3 instances.  (I conjecture that
might be similar to what Databricks does for dbfs:// under the hood ;)

Cheers,
-Paul



On Mon, Oct 15, 2018 at 4:28 PM Lu <[hidden email]> wrote:
>
> Hi Paul,
>
> Alluxio cache files to the first tier and then eviction may happen to put files to the other tiers(like in your case, the SSD tier). If the file size is bigger than the size of the first tier, we are not able to cache this file.
>
> If you want to cache your file in the SSD, you could set the SSD as the first tier.
>
> alluxio.worker.tieredstore.levels=1
>
> alluxio.worker.tieredstore.level0.alias=SSD
> alluxio.worker.tieredstore.level0.dirs.path=<your_path>
> alluxio.worker.tieredstore.level0.dirs.quota=<your_ssd_size>
> alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9
> alluxio.worker.tieredstore.level0.watermark.low.ratio=0.75
>
> Note that the reserved.ratio has been deprecated long ago, and the new configuration is watermark.high.ratio and watermark.low.ratio.
>
> If it still not work, please share us your worker logs and configuration settings(alluxio-site.properties).
>
> Thanks,
> Lu
>

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.