Is it possible to Run Alluxio for Spark with Databricks ?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Is it possible to Run Alluxio for Spark with Databricks ?

Akarsh Gupta
We use databricks for our spark workload, is it possible to configure Alluxio for the spark on databricks?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to Run Alluxio for Spark with Databricks ?

Bin Fan
Hi Akarsh,

Would you mind to share the problem you want to solve in your architecture here?

I think it is possible in theory, but I doubt it is straightforward. 

- Bin

On Tuesday, September 18, 2018 at 3:00:01 PM UTC-7, Akarsh Gupta wrote:
We use databricks for our spark workload, is it possible to configure Alluxio for the spark on databricks?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to Run Alluxio for Spark with Databricks ?

Akarsh Gupta
Hi Bin,

Thanks for your reply. The problem is  we run all our spark workloads on databricks, So now if I setup Alluxio in AWS EC2. How can i connect to Alluxio from Spark which runs on Databricks? does data bricks, how alluxio is  going to authenticate to spark running on databricks ? Is there any documentation ?

Thanks, Please let me know if you need any details ?
Best Regards,
Akarsh Gupta

On Wednesday, 19 September 2018 01:20:25 UTC-4, Bin Fan wrote:
Hi Akarsh,

Would you mind to share the problem you want to solve in your architecture here?

I think it is possible in theory, but I doubt it is straightforward. 

- Bin

On Tuesday, September 18, 2018 at 3:00:01 PM UTC-7, Akarsh Gupta wrote:
We use databricks for our spark workload, is it possible to configure Alluxio for the spark on databricks?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to Run Alluxio for Spark with Databricks ?

Gene Pang
Hi Akarsh,

I am not aware of someone running Alluxio on databricks so there will not be any documentation on this. However, it might be possible to install Alluxio on the same EC2 machines as the databricks machines, and point the Spark jobs to access Alluxio URIs. Since open source Spark can run on Alluxio, it is possible that Databricks Spark may also be able to use Alluxio.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to Run Alluxio for Spark with Databricks ?

Bin Fan
Regarding how to connect open source Apache Spark to Alluxio, here is the documentation: http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html

- Bin

On Tue, Oct 2, 2018 at 9:47 AM Gene Pang <[hidden email]> wrote:
Hi Akarsh,

I am not aware of someone running Alluxio on databricks so there will not be any documentation on this. However, it might be possible to install Alluxio on the same EC2 machines as the databricks machines, and point the Spark jobs to access Alluxio URIs. Since open source Spark can run on Alluxio, it is possible that Databricks Spark may also be able to use Alluxio.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.