Alluxio's latency performance

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Alluxio's latency performance

潘再余

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio's latency performance

David Zhu
Hi, Zaiyu 

Thanks for trying out Alluxio. 
As for the performance difference you observed between accessing files in Alluxio and accessing a local file system, it is large but also understandable. 
Alluxio has to contact the master for its metadata and involves round trips to the master and the worker to read a file, whereas reading from a local file system is much faster because everything is local. 

Also, could you describe your particular use case in more detail so we could understand it better?

Thanks, 

David



On Mon, Dec 3, 2018 at 7:03 PM 潘再余 <[hidden email]> wrote:

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio's latency performance

潘再余
Hi, David

Thanks for replying.

I understand what Alluxio cluster has to do to process the request. I originally planned to accelerate my distributed file system(not local) whose read latency is about 500us. So I expected that reading data cached in Alluxio can get lower latency, which proved to fail to meet my expectation.

Also, my cluster consists of a master and 3 worker. First, I write a 4KB file to UFS in THROUGH mode and 4KB block size which makes worker not cache the file. Second, I read this file in worker1 and the time is 12ms. So I take 12ms to be the latency of reading UFS. Third, I read this file in worker2 and the time is about 4ms. So I take 4ms to be the latency of reading remote memory. Finally, I read this file in worker2 again and the time is 3ms. So I take 3ms to be the latency of reading local memory. I have done these tests many times and gotten the same result. So, are my test results reasonable for Alluxio?

Regards.

Zaiyu


在 2018年12月5日星期三 UTC+8上午5:33:02,David Zhu写道:
Hi, Zaiyu 

Thanks for trying out Alluxio. 
As for the performance difference you observed between accessing files in Alluxio and accessing a local file system, it is large but also understandable. 
Alluxio has to contact the master for its metadata and involves round trips to the master and the worker to read a file, whereas reading from a local file system is much faster because everything is local. 

Also, could you describe your particular use case in more detail so we could understand it better?

Thanks, 

David



On Mon, Dec 3, 2018 at 7:03 PM 潘再余 <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="6PfPX1fGCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">panzai...@...> wrote:

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code <a href="https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;">https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="6PfPX1fGCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio's latency performance

Bin Fan
hi

are you mainly interested in reading/writing small files?

Currently Alluxio is more designed for serving file for bigdata workloads, with file sizes on the order
of at least tens of MBs in mind. Given the file size is 4KB, most of the time is spent on the roundtrip 
of metadata operations (e.g. create file inodes, connect to workers, acquire locks and etc), not on 
the I/O. In this case, serving data in memory or not may not make much difference unless the benchmark
is very carefully controlled.

- Bin


On Tuesday, December 4, 2018 at 7:21:50 PM UTC-8, 潘再余 wrote:
Hi, David

Thanks for replying.

I understand what Alluxio cluster has to do to process the request. I originally planned to accelerate my distributed file system(not local) whose read latency is about 500us. So I expected that reading data cached in Alluxio can get lower latency, which proved to fail to meet my expectation.

Also, my cluster consists of a master and 3 worker. First, I write a 4KB file to UFS in THROUGH mode and 4KB block size which makes worker not cache the file. Second, I read this file in worker1 and the time is 12ms. So I take 12ms to be the latency of reading UFS. Third, I read this file in worker2 and the time is about 4ms. So I take 4ms to be the latency of reading remote memory. Finally, I read this file in worker2 again and the time is 3ms. So I take 3ms to be the latency of reading local memory. I have done these tests many times and gotten the same result. So, are my test results reasonable for Alluxio?

Regards.

Zaiyu


在 2018年12月5日星期三 UTC+8上午5:33:02,David Zhu写道:
Hi, Zaiyu 

Thanks for trying out Alluxio. 
As for the performance difference you observed between accessing files in Alluxio and accessing a local file system, it is large but also understandable. 
Alluxio has to contact the master for its metadata and involves round trips to the master and the worker to read a file, whereas reading from a local file system is much faster because everything is local. 

Also, could you describe your particular use case in more detail so we could understand it better?

Thanks, 

David



On Mon, Dec 3, 2018 at 7:03 PM 潘再余 <[hidden email]> wrote:

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code <a href="https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;">https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio's latency performance

Manish Malhotra
Its an interesting thread, I also want to ask another question on the similar line.

With my limited knowledge of Alluxio till now.

As we are saying, if we have small files then Alluxio's performance might be having issue, and rightly so as same as Hadoop Namenode, because NN also maintains the metadata of namespace and if its too much, then NN start having problem and performance also degrade.

Now, if the similar problem can happen in Alluxio, then even though we are not using HDFS storage as underlying storage, we can hit the metadata issue as well.
So, for that may be better option is to keep metadata in separate layer or distribute the metadata in the worker node as data?

Regards,
- Manish

On Friday, December 7, 2018 at 12:06:43 AM UTC-8, Bin Fan wrote:
hi

are you mainly interested in reading/writing small files?

Currently Alluxio is more designed for serving file for bigdata workloads, with file sizes on the order
of at least tens of MBs in mind. Given the file size is 4KB, most of the time is spent on the roundtrip 
of metadata operations (e.g. create file inodes, connect to workers, acquire locks and etc), not on 
the I/O. In this case, serving data in memory or not may not make much difference unless the benchmark
is very carefully controlled.

- Bin


On Tuesday, December 4, 2018 at 7:21:50 PM UTC-8, 潘再余 wrote:
Hi, David

Thanks for replying.

I understand what Alluxio cluster has to do to process the request. I originally planned to accelerate my distributed file system(not local) whose read latency is about 500us. So I expected that reading data cached in Alluxio can get lower latency, which proved to fail to meet my expectation.

Also, my cluster consists of a master and 3 worker. First, I write a 4KB file to UFS in THROUGH mode and 4KB block size which makes worker not cache the file. Second, I read this file in worker1 and the time is 12ms. So I take 12ms to be the latency of reading UFS. Third, I read this file in worker2 and the time is about 4ms. So I take 4ms to be the latency of reading remote memory. Finally, I read this file in worker2 again and the time is 3ms. So I take 3ms to be the latency of reading local memory. I have done these tests many times and gotten the same result. So, are my test results reasonable for Alluxio?

Regards.

Zaiyu


在 2018年12月5日星期三 UTC+8上午5:33:02,David Zhu写道:
Hi, Zaiyu 

Thanks for trying out Alluxio. 
As for the performance difference you observed between accessing files in Alluxio and accessing a local file system, it is large but also understandable. 
Alluxio has to contact the master for its metadata and involves round trips to the master and the worker to read a file, whereas reading from a local file system is much faster because everything is local. 

Also, could you describe your particular use case in more detail so we could understand it better?

Thanks, 

David



On Mon, Dec 3, 2018 at 7:03 PM 潘再余 <[hidden email]> wrote:

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code <a href="https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;">https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio's latency performance

潘再余
In reply to this post by Bin Fan
Hi.

Sorry for so long to reply. And I understand now.
Maybe I should improve my architecture design to adapt to my demand.

Thanks very much.

Pan

在 2018年12月7日星期五 UTC+8下午4:06:43,Bin Fan写道:
hi

are you mainly interested in reading/writing small files?

Currently Alluxio is more designed for serving file for bigdata workloads, with file sizes on the order
of at least tens of MBs in mind. Given the file size is 4KB, most of the time is spent on the roundtrip 
of metadata operations (e.g. create file inodes, connect to workers, acquire locks and etc), not on 
the I/O. In this case, serving data in memory or not may not make much difference unless the benchmark
is very carefully controlled.

- Bin


On Tuesday, December 4, 2018 at 7:21:50 PM UTC-8, 潘再余 wrote:
Hi, David

Thanks for replying.

I understand what Alluxio cluster has to do to process the request. I originally planned to accelerate my distributed file system(not local) whose read latency is about 500us. So I expected that reading data cached in Alluxio can get lower latency, which proved to fail to meet my expectation.

Also, my cluster consists of a master and 3 worker. First, I write a 4KB file to UFS in THROUGH mode and 4KB block size which makes worker not cache the file. Second, I read this file in worker1 and the time is 12ms. So I take 12ms to be the latency of reading UFS. Third, I read this file in worker2 and the time is about 4ms. So I take 4ms to be the latency of reading remote memory. Finally, I read this file in worker2 again and the time is 3ms. So I take 3ms to be the latency of reading local memory. I have done these tests many times and gotten the same result. So, are my test results reasonable for Alluxio?

Regards.

Zaiyu


在 2018年12月5日星期三 UTC+8上午5:33:02,David Zhu写道:
Hi, Zaiyu 

Thanks for trying out Alluxio. 
As for the performance difference you observed between accessing files in Alluxio and accessing a local file system, it is large but also understandable. 
Alluxio has to contact the master for its metadata and involves round trips to the master and the worker to read a file, whereas reading from a local file system is much faster because everything is local. 

Also, could you describe your particular use case in more detail so we could understand it better?

Thanks, 

David



On Mon, Dec 3, 2018 at 7:03 PM 潘再余 <[hidden email]> wrote:

Hi all,

I start using Alluxio on version 1.8.1. Before deploying it, I'd like to get Alluxio's latency performance. And I used the code <a href="https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fmaster%2Fexamples%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fexamples%2FPerformance.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFwQAv6cFV1HZOaLVRgEtxsDAvEyg&#39;;return true;">https://github.com/Alluxio/alluxio/blob/master/examples/src/main/java/alluxio/examples/Performance.java.

So I set up a test environment with 4 servers, of which one is master and three are workers. The UFS was a HDFS cluster with the same 4 servers. And I measured the latency of Alluxio by the time of reading a 4KB block.

The test result is that the time of reading data in UFS(meaning it hasn't been cached in Alluxio) is about 12ms and the time of reading data in Alluxio(meaning it has been cached in Alluxio) is about 3ms. 

It seems that the cache of Alluxio does reduce the latency. What bothers me is that the two latency are too high to believe. Because the same latency of local file system (without Alluxio, tested by fio) is 200us. The overhead of getting metadata and transferring data in Alluxio seems too high.

Furthermore, the time result of reading test  is mainly composed of two parts - openFile() and read(). So I tested them separately. With the data cached, the time of openFile() is 2ms and the time of read() is 1~2ms.

I'd like to ask whether my test results match Alluxio's latency performance, or maybe I got some configuration problems.

Thank you all.
Zaiyu Pan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.