Can I checkpoint spark RDD to Alluxio

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Can I checkpoint spark RDD to Alluxio

Wanchun Wang
hello folks in Alluxio,

I spoke with Haoyuan at Spark Summit and started to evaluate your product shortly after that. Great product, looking forward to use it to cache accumulated data in our next generation data platform.


I have a quick question, we use Spark streaming a lot, can I use Alluxio as checkpoint destination?

something like this:
val ssc = new StreamingContext(sc, Duration.apply(3000))
sss.checkpoint("alluxio://host:19998/checkpointdir")

If this is possible, I plan to pick the data from checkpoint dir and  use the data in another Spark app, will it work?

Thanks
Wanchun Wang
VipshopUS

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I checkpoint spark RDD to Alluxio

Pei Sun
Yes, this should work.  Qunar used this and published a blog (in Chinese) about this. 

On Fri, Jul 8, 2016 at 4:05 PM, Wanchun Wang <[hidden email]> wrote:
hello folks in Alluxio,

I spoke with Haoyuan at Spark Summit and started to evaluate your product shortly after that. Great product, looking forward to use it to cache accumulated data in our next generation data platform.


I have a quick question, we use Spark streaming a lot, can I use Alluxio as checkpoint destination?

something like this:
val ssc = new StreamingContext(sc, Duration.apply(3000))
sss.checkpoint("alluxio://host:19998/checkpointdir")

If this is possible, I plan to pick the data from checkpoint dir and  use the data in another Spark app, will it work?

Thanks
Wanchun Wang
VipshopUS

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I checkpoint spark RDD to Alluxio

Wanchun Wang
good to know. I will give it a spin. 

Thanks

Wanchun

Sent from my iPhone

On Jul 8, 2016, at 4:11 PM, Pei Sun <[hidden email]> wrote:

Yes, this should work.  Qunar used this and published a blog (in Chinese) about this. 

On Fri, Jul 8, 2016 at 4:05 PM, Wanchun Wang <[hidden email]> wrote:
hello folks in Alluxio,

I spoke with Haoyuan at Spark Summit and started to evaluate your product shortly after that. Great product, looking forward to use it to cache accumulated data in our next generation data platform.


I have a quick question, we use Spark streaming a lot, can I use Alluxio as checkpoint destination?

something like this:
val ssc = new StreamingContext(sc, Duration.apply(3000))
sss.checkpoint("alluxio://host:19998/checkpointdir")

If this is possible, I plan to pick the data from checkpoint dir and  use the data in another Spark app, will it work?

Thanks
Wanchun Wang
VipshopUS

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I checkpoint spark RDD to Alluxio

Pei Sun
Hi Wanchun,
   How did it go? We are interested in knowing how it worked for you?
Thank you
Pei

On Fri, Jul 8, 2016 at 4:22 PM, Wanchun.wang <[hidden email]> wrote:
good to know. I will give it a spin. 

Thanks

Wanchun

Sent from my iPhone

On Jul 8, 2016, at 4:11 PM, Pei Sun <[hidden email]> wrote:

Yes, this should work.  Qunar used this and published a blog (in Chinese) about this. 

On Fri, Jul 8, 2016 at 4:05 PM, Wanchun Wang <[hidden email]> wrote:
hello folks in Alluxio,

I spoke with Haoyuan at Spark Summit and started to evaluate your product shortly after that. Great product, looking forward to use it to cache accumulated data in our next generation data platform.


I have a quick question, we use Spark streaming a lot, can I use Alluxio as checkpoint destination?

something like this:
val ssc = new StreamingContext(sc, Duration.apply(3000))
sss.checkpoint("alluxio://host:19998/checkpointdir")

If this is possible, I plan to pick the data from checkpoint dir and  use the data in another Spark app, will it work?

Thanks
Wanchun Wang
VipshopUS

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun



--
Pei Sun

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.