Issue for downloading underfs throught Alluxio proxy

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue for downloading underfs throught Alluxio proxy

Autaa François
Alluxio 1.7.1-hadoop-2.8
Mesos 1.4.1
Ceph Luminous for DFS and Object Storage with S3 interface 
Centos 7.3.1611
JDK 1.8.0_131

Hi

I've some issues with the proxy. I've been working with boto3 to download file using S3 over the s3 proxy.

But when I check the proxy.out I've some severe issues :

PM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
SEVERE
: An I/O error has occurred while writing a response message entity to the container output stream.
org
.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
    at org
.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org
.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
    at org
.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:697)
    at org
.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:432)
    at org
.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:422)
    at org
.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:320)
    at org
.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org
.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org
.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org
.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org
.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org
.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org
.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
    at org
.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
    at org
.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
    at org
.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
    at org
.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
    at org
.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
    at org
.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
    at org
.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org
.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
    at org
.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org
.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at org
.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org
.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org
.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org
.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org
.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org
.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org
.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org
.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org
.eclipse.jetty.server.Server.handle(Server.java:499)
    at org
.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org
.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org
.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org
.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org
.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java
.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.io.EofException
    at org
.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
    at org
.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
    at org
.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
    at org
.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)
    at org
.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:708)
    at org
.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
    at org
.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
    at org
.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:480)
    at org
.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)
    at org
.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
    at org
.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
    at org
.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140)
    at org
.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:355)
    at org
.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
    at org
.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at org
.glassfish.jersey.message.internal.ReaderWriter.writeTo(ReaderWriter.java:116)
    at org
.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:79)
    at org
.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:105)
    at org
.glassfish.jersey.message.internal.InputStreamProvider.writeTo(InputStreamProvider.java:60)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org
.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org
.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org
.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
   
... 38 more
Caused by: java.io.IOException: Broken pipe
    at sun
.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun
.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun
.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun
.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun
.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
    at org
.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
   
... 63 more

This cause some slow down for downloading the file and in threaded mode (python/boto3) make my execution fail .
Anyone else have this kind of issue ?

Regards

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue for downloading underfs throught Alluxio proxy

Autaa François
This only happen on big file file more than 100mb

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue for downloading underfs throught Alluxio proxy

Autaa François
In reply to this post by Autaa François
I've also tried to use the REST API interface to see if it's a proxy side issue and the curl command take a normal time to download my files. I've no idea whats going wrong with boto3 and the proxy

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue for downloading underfs throught Alluxio proxy

Gene Pang
Hi,

Is there a proxy.log file too? Is there more information in that log file?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue for downloading underfs throught Alluxio proxy

Autaa François
Nothing more interesting into the log files. It juste says the proxy have been launched.

I've managed to run my test by using low level python format read function. this one works and the boto3 version does not works.

Regards

On Thursday, May 24, 2018 at 4:43:53 PM UTC+2, Gene Pang wrote:
Hi,

Is there a proxy.log file too? Is there more information in that log file?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue for downloading underfs throught Alluxio proxy

Gene Pang
Thanks for the update!

If the boto3 version does not work, could you add a jira ticket describing the issue?

Thanks,

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.