Partition the data for each worker

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Partition the data for each worker

Md Mahbub Alam
Hi All,

When we write data into alluxio, It automatically distributes data among workers. My first question is "what is the technique used by alluxio to distribute data?". My second question is "If I want to partition the data for each worker manually, how can I do that? Any suggestion?

thanks
Alam

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Partition the data for each worker

Gene Pang
Hi Alam,

There are different write policies built into Alluxio. The default is LocalFirst, which will write the block on the local worker.

There are other policies available, and you can read more about them here:

http://www.alluxio.org/docs/1.7/en/Clients-Alluxio-Java.html#location-policy

After the file is written, I don't think there is a simple way to re-partition the data manually.

What is the use case in which you need such specific partitioning control?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.