how to configure to run spark on yarn with alluxio without change any code?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how to configure to run spark on yarn with alluxio without change any code?

alexbaijw
I have set up a alluxio cluster, and each dataNode has one worker on it. And I also set the hdfs as the alluxio storage backend. If I want to run a spark job on yarn in yarn-client mode, how can I use the alluxio cluster to speed up my job?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to configure to run spark on yarn with alluxio without change any code?

Gene Pang
Hi Alex,

Here is some documentation for running Spark with Alluxio: http://www.alluxio.org/docs/1.7/en/Running-Spark-on-Alluxio.html

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to configure to run spark on yarn with alluxio without change any code?

alexbaijw
Thanks, I had already read the documents. Since I do not have a spark setup, I just use the spark jars on the hdfs by configuration ”spark.yarn.jars“.  I just need to add the alluxio-client jar to the classpath of my spark-sql app? 
在 2018年5月29日星期二 UTC+8下午11:07:39,Gene Pang写道:
Hi Alex,

Here is some documentation for running Spark with Alluxio: <a href="http://www.alluxio.org/docs/1.7/en/Running-Spark-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FRunning-Spark-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNELp9ew9aZ6_YVJu04xpjOmunw21w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FRunning-Spark-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNELp9ew9aZ6_YVJu04xpjOmunw21w&#39;;return true;">http://www.alluxio.org/docs/1.7/en/Running-Spark-on-Alluxio.html

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.