Question about multi-master deployment

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about multi-master deployment

shangrong11
Luckily to have a place to ask questions. I have been trying to deploy Alluxio on a cluster for months. I came across several hard problems to
solve. 

The first important problem is on high-availability deployment. I deployed two masters, and a zookeeper server on each Alluxio master/worker node. At a
time, only one master can work, and the other master takes over only when the leader master fails. My question is: Is there a deployment that multiple
masters can work together to provide metadata service? I think it is quite important for large-scale cluster with for example thousands of clients. If
thousands of threads try to access IO, may the only leader master becomes a bottleneck? I wish there is a solution for this problem, but did not find an
answer yet. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Question about multi-master deployment

binfan
Administrator
Hi,

- Currently in multi-master deployment of Alluxio, only the lead master will serve users.

- Note that, only the filesystem metadata operations are served by the master while
the real I/O transfer is served by the workers, concurrently.
Alluxio dev team has made a lot of effort to make master highly concurrent to handle client requests.
I have definitely seen the use case with thousands of clients accessing the lead master.

- Bin

On Sunday, July 22, 2018 at 7:29:35 PM UTC-7, [hidden email] wrote:
Luckily to have a place to ask questions. I have been trying to deploy Alluxio on a cluster for months. I came across several hard problems to
solve. 

The first important problem is on high-availability deployment. I deployed two masters, and a zookeeper server on each Alluxio master/worker node. At a
time, only one master can work, and the other master takes over only when the leader master fails. My question is: Is there a deployment that multiple
masters can work together to provide metadata service? I think it is quite important for large-scale cluster with for example thousands of clients. If
thousands of threads try to access IO, may the only leader master becomes a bottleneck? I wish there is a solution for this problem, but did not find an
answer yet. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.