ASYNC_THROUGH write type is experimental?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ASYNC_THROUGH write type is experimental?

Hector Zhang
I want to use feature in product environment, but I found this an experimental feature. I am wondering how to understand it, is there any known fault?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ASYNC_THROUGH write type is experimental?

Gene Pang
Hi Hector,

I believe it is experimental because it has some limitations. The main limitation is that all the blocks of the file must be resident on the same worker in order for the async process to persist the file into the UFS. Otherwise, it will not be persisted. If you are using the LocalFirst write policy, that scenario may be highly likely. However, it is not guaranteed.

Could you briefly describe the use case you want to use ASYNC_THROUGH for?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ASYNC_THROUGH write type is experimental?

Hector Zhang
The scenario is like this, I am using alluxio as uniform file system of structured big data ETL platform, such as create a table in hive meta with location points to a path in alluxio, the ETL task generated a partition each day and write data to alluxio. I want the data of the partition written to alluxio be persisted when it is saved to alluxio.

About this limitation, if the file is too big to be saved on any worker, then ASYNC_THROUGH scheme will cause the file not be persisted?



On Thursday, May 24, 2018 at 10:31:06 PM UTC+8, Gene Pang wrote:
Hi Hector,

I believe it is experimental because it has some limitations. The main limitation is that all the blocks of the file must be resident on the same worker in order for the async process to persist the file into the UFS. Otherwise, it will not be persisted. If you are using the LocalFirst write policy, that scenario may be highly likely. However, it is not guaranteed.

Could you briefly describe the use case you want to use ASYNC_THROUGH for?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ASYNC_THROUGH write type is experimental?

Gene Pang
Yes, currently, it will not be persisted. You could write the file as CACHE_THROUGH to work around the limitation.

Thanks,
Gene

alluxio.com | alluxio.org powered by Alluxio

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.