Does Alluxio provide block replication?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Does Alluxio provide block replication?

wayasxxx

Hi,

Does Alluxio provide any features like HDFS replication? 
In another word, is it possible for one block exist in more than one worker at the same time?  
If it is, could you tell me the source code address about it. I want to know the technique detail.

Thanks,

Anyang

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

Gene Pang
Hi Anyang,

An Alluxio block can be in more than 1 worker at the same time. This is achieved by "lazy" replication, meaning, if a block is required on a different machine because of a task on that machine, Alluxio can copy that block to that machine, thus increasing the replication factor.

The passive caching ability performs this "lazy" replication.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

ZHOU Yuan
Hi Gene,

Thanks for the detail explanation. 

It looks like in this blog, Alluxio can provide 2x replication when handling writes. Is this a feature in Alluxio enterprise edition only? 





Sincerely, Yuan

On Fri, Apr 27, 2018 at 10:14 PM, Gene Pang <[hidden email]> wrote:
Hi Anyang,

An Alluxio block can be in more than 1 worker at the same time. This is achieved by "lazy" replication, meaning, if a block is required on a different machine because of a task on that machine, Alluxio can copy that block to that machine, thus increasing the replication factor.

The passive caching ability performs this "lazy" replication.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

Gene Pang
Hi Yuan,

Yes, in the enterprise edition, there is a feature called fast durable writes, which writes to multiple replicas, and asynchronously writes to the UFS in the background. Is that something you would be interested in trying?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

Gene Pang
Hi Yuan,

Were we able to answer your questions? Do you have any follow up questions on replication?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

ZHOU Yuan
Hi Gene,

Yeah that answers my question, thanks for the explanation. 

Sincerely, Yuan


On Thu, May 24, 2018 at 10:46 PM Gene Pang <[hidden email]> wrote:
Hi Yuan,

Were we able to answer your questions? Do you have any follow up questions on replication?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Does Alluxio provide block replication?

Gene Pang
I'm glad we could help!

-Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.