File not fully loaded in cache

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

File not fully loaded in cache

Pascal Gillet
Alluxio 1.7.1-hadoop-2.8
Mesos 1.4.1
Ceph Luminous for DFS and Object Storage with S3 interface 
Centos 7.3.1611
JDK 1.8.0_131
Spark 2.2.1

Hi,

I try to run a simple Spark Job in a Mesos cluster to test Alluxio as described in https://www.alluxio.com/blog/getting-started-with-alluxio-and-spark.

Whenever I use ./bin/alluxio fs load /path/to/bigTextFile or an Alluxio URL in my Spark job to fetch a big text file from the S3 backend, the said file is not fully loaded into memory, while there is plenty of room across my workers:

sc = SparkContext()
file = sc.textFile('alluxio://path/to/bigTextFile')
print(file.count())

Why is that?
Do I miss some obvious Alluxio configuration?

Thanks,

Pascal GILLET


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Gene Pang
Hi Pascal,

I have a few questions about your environment:

- Does this spark job succeed?
- Is any portion of the file in memory?
- Is Alluxio deployed on mesos? which containerizer?
- How many Alluxio workers?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Gene Pang
Hi Pascal,

I think what is happening is that the loading is writing to only a single machine. Since a single worker does not have space for the whole file, it does not fully get loaded into memory.

This can be controlled by the write location policy. The default is LocalFirstPolicy, but you can change it to a different policy. More documentation about the location policy is here: https://www.alluxio.org/docs/1.7/en/Clients-Alluxio-Java.html#location-policy

Thanks,
Gene

On Fri, Apr 27, 2018 at 7:57 AM, Pascal Gillet <[hidden email]> wrote:
Other details:
- My file is never totally loaded in memory even with ./bin/alluxio fs load /path/to/bigTextFile
- The worker memory size is 2 GB * 6 ~ 8 workers: 12 GB min ~ 16 GB max total 

2018-04-27 16:48 GMT+02:00 Pascal Gillet <[hidden email]>:
Hi Gene,

- Yes, the Spark job succeeds but it takes as much time as if it reads the file directly from the S3 backend.

- Yes, I have a 8GB file that is never totally loaded in memory: from 25% to 50%, if I run the job multiple times, and I can see that the data in memory sometimes moves from workers to others.

- Alluxio is deployed with High Availability with ZooKeeper but not through Mesos: I have 3 Alluxio masters on separate servers, 3 Mesos masters, and the Alluxio workers are also the Mesos agents (whose resources are partitioned for Mesos and Alluxio)

- My Spark job is packaged in a Docker container executed through Mesos: I mounted the Alluxio client jar in the Spark classpath, and I also had to mount /mnt/ramdisk, otherwise Spark complains about FileNotFoundExceptions 

- 6 ~ 8 Alluxio Workers (Mesos agents)

Thanks,
Pascal

2018-04-27 16:20 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

I have a few questions about your environment:

- Does this spark job succeed?
- Is any portion of the file in memory?
- Is Alluxio deployed on mesos? which containerizer?
- How many Alluxio workers?

Thanks,
Gene

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/dQYdzrgBb6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Pascal Gillet
Hi Gene,

Yes, that was it!
The property was under my eyes all the time...

Thank you very much,

Pascal

2018-04-27 17:06 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

I think what is happening is that the loading is writing to only a single machine. Since a single worker does not have space for the whole file, it does not fully get loaded into memory.

This can be controlled by the write location policy. The default is LocalFirstPolicy, but you can change it to a different policy. More documentation about the location policy is here: https://www.alluxio.org/docs/1.7/en/Clients-Alluxio-Java.html#location-policy

Thanks,
Gene

On Fri, Apr 27, 2018 at 7:57 AM, Pascal Gillet <[hidden email]> wrote:
Other details:
- My file is never totally loaded in memory even with ./bin/alluxio fs load /path/to/bigTextFile
- The worker memory size is 2 GB * 6 ~ 8 workers: 12 GB min ~ 16 GB max total 

2018-04-27 16:48 GMT+02:00 Pascal Gillet <[hidden email]>:
Hi Gene,

- Yes, the Spark job succeeds but it takes as much time as if it reads the file directly from the S3 backend.

- Yes, I have a 8GB file that is never totally loaded in memory: from 25% to 50%, if I run the job multiple times, and I can see that the data in memory sometimes moves from workers to others.

- Alluxio is deployed with High Availability with ZooKeeper but not through Mesos: I have 3 Alluxio masters on separate servers, 3 Mesos masters, and the Alluxio workers are also the Mesos agents (whose resources are partitioned for Mesos and Alluxio)

- My Spark job is packaged in a Docker container executed through Mesos: I mounted the Alluxio client jar in the Spark classpath, and I also had to mount /mnt/ramdisk, otherwise Spark complains about FileNotFoundExceptions 

- 6 ~ 8 Alluxio Workers (Mesos agents)

Thanks,
Pascal

2018-04-27 16:20 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

I have a few questions about your environment:

- Does this spark job succeed?
- Is any portion of the file in memory?
- Is Alluxio deployed on mesos? which containerizer?
- How many Alluxio workers?

Thanks,
Gene

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/dQYdzrgBb6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.




--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Pascal Gillet
Hi Gene,

I also started the Alluxio HTTP proxy on all my workers to interact with Alluxio using the S3 API with a boto client.
I added the alluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy in the configuration file on all my workers.
As the proxy becomes a client itself, I would have expected that this user specific property would be applicable and that all the files downloaded through the proxy would be completely loaded in memory and scattered across the workers.
But only some files are loaded on the proxy host.

Did I miss another configuration?

Thanks,

Pascal

2018-05-01 11:10 GMT+02:00 Pascal Gillet <[hidden email]>:
Hi Gene,

Yes, that was it!
The property was under my eyes all the time...

Thank you very much,

Pascal

2018-04-27 17:06 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

I think what is happening is that the loading is writing to only a single machine. Since a single worker does not have space for the whole file, it does not fully get loaded into memory.

This can be controlled by the write location policy. The default is LocalFirstPolicy, but you can change it to a different policy. More documentation about the location policy is here: https://www.alluxio.org/docs/1.7/en/Clients-Alluxio-Java.html#location-policy

Thanks,
Gene

On Fri, Apr 27, 2018 at 7:57 AM, Pascal Gillet <[hidden email]> wrote:
Other details:
- My file is never totally loaded in memory even with ./bin/alluxio fs load /path/to/bigTextFile
- The worker memory size is 2 GB * 6 ~ 8 workers: 12 GB min ~ 16 GB max total 

2018-04-27 16:48 GMT+02:00 Pascal Gillet <[hidden email]>:
Hi Gene,

- Yes, the Spark job succeeds but it takes as much time as if it reads the file directly from the S3 backend.

- Yes, I have a 8GB file that is never totally loaded in memory: from 25% to 50%, if I run the job multiple times, and I can see that the data in memory sometimes moves from workers to others.

- Alluxio is deployed with High Availability with ZooKeeper but not through Mesos: I have 3 Alluxio masters on separate servers, 3 Mesos masters, and the Alluxio workers are also the Mesos agents (whose resources are partitioned for Mesos and Alluxio)

- My Spark job is packaged in a Docker container executed through Mesos: I mounted the Alluxio client jar in the Spark classpath, and I also had to mount /mnt/ramdisk, otherwise Spark complains about FileNotFoundExceptions 

- 6 ~ 8 Alluxio Workers (Mesos agents)

Thanks,
Pascal

2018-04-27 16:20 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

I have a few questions about your environment:

- Does this spark job succeed?
- Is any portion of the file in memory?
- Is Alluxio deployed on mesos? which containerizer?
- How many Alluxio workers?

Thanks,
Gene

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/dQYdzrgBb6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.





--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Gene Pang
Hi Pascal,

If you use that configuration parameter on the client itself, does that change the behavior?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Gene Pang
Hi Pascal,

Were you able to resolve your issue?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: File not fully loaded in cache

Pascal Gillet
Hi Gene,

Thank you for coming back to me.
Actually, no, because I don't know how to/can't pass the alluxio.user.file.write.location.policy.class property from my boto3 client (I do not use the Python Alluxio client).

Thanks,
Pascal

2018-05-24 16:47 GMT+02:00 Gene Pang <[hidden email]>:
Hi Pascal,

Were you able to resolve your issue?

Thanks,
Gene

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/dQYdzrgBb6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.