how it works that a mapreduce job on a alluxio cluster?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how it works that a mapreduce job on a alluxio cluster?

wgcris
i recently start to research the alluxio framework , i set up the test environment ,
a hadoop cluster, including 9 datanode and 1 namenode;
a alluxio cluster, 1 master,9 worker, per worker mem size=20GB; 
i did two experiments 
1.put 9GB file to HDFS,and run MR twice ,the runtime=9m25.668s,10m22.945s
2, load 9GB from HDFS to alluxio ;run MR twice, the runtime=10m39.473s, 9m3.069s
why it turn out to be this results?
who can tell me the difference when runing MR on  HDFS and Alluxio ?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how it works that a mapreduce job on a alluxio cluster?

Calvin Jia
Hi,

I think the data will be in buffer cache (OS managed memory cache) for the HDFS case, so the comparison will be very similar. You can try dropping the buffer cache on your machines after putting the file into HDFS. Also, what kind of workload are you running? It seems like it is not I/O bound. In general, Alluxio will only help with the I/O portion of the workload, which is 1 GB/node if you are reading the data once. If your disk has a throughput of 100 MB/s, you will only be saving 10 or so seconds.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how it works that a mapreduce job on a alluxio cluster?

Calvin Jia
Hi,

Were any of my suggestions helpful?

Thanks,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.