Keeping blocks in tier 1

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Keeping blocks in tier 1

abellina
Hello

I am looking to figure out if it's possible for blocks to remain in tier 1 while they are being copied to tier 0, rather than moved. The reason I ask is I'd like to load up a pool of fast SSDs with all of my data, or most of it. Then let the tier 0 <-> tier 1 promotion to help with especially hot blocks. 

Currently I see that tier 1 never fills up, even though I cycle through my queries (Spark in this case) many times. Oddly, I do not see eviction metrics increasing in the workers, so not sure why these blocks get promoted and then disappear, without altering a metric. 

Note, if the cache is single tiered then this can be accomplished, but you loose having RAM as a tier 0.

Thanks

Alessandro

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Keeping blocks in tier 1

Calvin Jia
Hi,

Alluxio currently does not support duplication of blocks with a worker's local storage. However this has been requested a few times and could be an interesting feature, especially when lower tiers have much more space and the remote storage is much slower than the lower tier.

Could you open a JIRA ticket with details on how you would like the duplication between tiers to work for your use case?

Thanks,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.