Alluxio netty class load issue

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Alluxio netty class load issue

Jais Sebastian
We are using Alluxio 1.8.0 with Spark 2.2.0. During file write from our Java client API we get below error. This code was working for us before and after we rebuilt the code ( there is no change in the code as well ), we started getting below error


fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] alluxio.AbstractClient                    257 : Alluxio client (version 1.8.0) is trying to connect with BlockMasterClient @ <MASTER-HOST>/<MASTER>:31398
2018-10-18 03:13:36.356 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] a.s.a.TransportProviderUtils               47 : Impersonation: subject: Subject:
Principal: hdfs

2018-10-18 03:13:36.356 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] a.s.a.TransportProviderUtils               57 : Impersonation: configured: _HDFS_USER_
2018-10-18 03:13:36.357 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] a.s.a.TransportProviderUtils               73 : Impersonation: hdfsUser: hdfs impersonationUser: hdfs
2018-10-18 03:13:36.357 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       261 : opening transport org.apache.thrift.transport.TSaslClientTransport@5b477a55
2018-10-18 03:13:36.358 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.t.transport.TSaslClientTransport       96 : Sending mechanism name PLAIN and initial response of length 20
2018-10-18 03:13:36.358 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       162 : CLIENT: Writing message with status START and payload length 5
2018-10-18 03:13:36.359 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       162 : CLIENT: Writing message with status COMPLETE and payload length 20
2018-10-18 03:13:36.359 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       273 : CLIENT: Start message handled
2018-10-18 03:13:36.359 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       296 : CLIENT: Main negotiation loop complete
2018-10-18 03:13:36.359 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       306 : CLIENT: SASL Client receiving last message
2018-10-18 03:13:36.360 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       206 : CLIENT: Received message with status COMPLETE and payload length 0
2018-10-18 03:13:36.360  INFO 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] alluxio.AbstractClient                    264 : Client registered with BlockMasterClient @ <MASTER-HOST>/<MASTER>:31398
2018-10-18 03:13:36.361 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       498 : writing data length: 52
2018-10-18 03:13:36.362 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 45
2018-10-18 03:13:36.370 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       498 : writing data length: 52
2018-10-18 03:13:36.374 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 2199
2018-10-18 03:13:36.419 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.c.loader.WebappClassLoaderBase        179 :     findClass(io.netty.util.internal.__matchers__.alluxio.network.protocol.RPCMessageMatcher)
2018-10-18 03:13:36.419 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] o.a.c.loader.WebappClassLoaderBase        179 :     --> Returning ClassNotFoundException
2018-10-18 03:13:36.422 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] i.JavassistTypeParameterMatcherGenerator   76 : Generated: io.netty.util.internal.__matchers__.alluxio.network.protocol.RPCMessageMatcher
2018-10-18 03:13:36.434 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] u.d.--.log.MethodLogger              78 : END - .HdfsWriter.writeFile :: END TIME - 1539832416434
2018-10-18 03:13:36.435 DEBUG 1  --- [6b90a4fd-ab3a-4859-82d4-25c908f4754a, dev2] [nio-9828-exec-7] u.d.--.log.MethodLogger              78 : END - --.DataFileServiceImpl.ingest :: END TIME - 1539832416435
2
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.Native
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:978)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

Below are the list of netty jars in the class path 


netty-3.9.9.Final.jar
netty-all-4.0.53.Final.jar
netty-buffer-4.0.27.Final.jar
netty-codec-4.0.27.Final.jar
netty-codec-http-4.0.27.Final.jar
netty-common-4.0.27.Final.jar
netty-handler-4.0.27.Final.jar
netty-transport-4.0.27.Final.jar
netty-transport-native-epoll-4.0.27.Final.jar
rxnetty-0.4.9.jar
rxnetty-contexts-0.4.9.jar
rxnetty-servo-0.4.9.jar

With same libraries our older builds are working fine.

What could be the reason for this issue?

Regards,
Jais

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio netty class load issue

Gene Pang
Hi Jais,

What was the difference between when it was working and and not working? Were you always on the same 1.8.0 version? Is there more to the stacktrace for "org.springframework.web.util.NestedServletException"? That doesn't look Alluxio related, but maybe there is more info in the stack. Or, is there some Alluxio error anywhere, in the client, master or worker logs?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio netty class load issue

Jais Sebastian
There is no error on Alluxio side.
Below is the complete stack trace

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.Native
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:978)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter.doFilter(KeycloakSecurityContextRequestFilter.java:79)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter.doFilter(KeycloakAuthenticatedActionsFilter.java:82)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at --.filter.CORSFilter.doFilter(CORSFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at --.config.security.AppIdEnrichFilter.doFilterInternal(AppIdEnrichFilter.java:53)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticationProcessingFilter.successfulAuthentication(KeycloakAuthenticationProcessingFilter.java:206)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:240)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter.doFilter(KeycloakPreAuthActionsFilter.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at --.filter.LogFilter.doFilter(LogFilter.java:43)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.Native
at io.netty.channel.epoll.EpollEventArray.<clinit>(EpollEventArray.java:40)
at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:65)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
at alluxio.util.network.NettyUtils.createEventLoop(NettyUtils.java:72)
at alluxio.network.netty.NettyClient.<clinit>(NettyClient.java:54)
at alluxio.client.file.FileSystemContext.acquireNettyChannel(FileSystemContext.java:338)
at alluxio.client.block.stream.NettyPacketWriter.create(NettyPacketWriter.java:127)
at alluxio.client.block.stream.UnderFileSystemFileOutStream.create(UnderFileSystemFileOutStream.java:41)
at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:110)
at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:144)
at alluxio.hadoop.AbstractFileSystem.create(AbstractFileSystem.java:179)
at alluxio.hadoop.FileSystem.create(FileSystem.java:27)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:816)
at --.--.writer.HdfsWriter.writeFile(HdfsWriter.java:120)
at --.--.writer.HdfsWriter.writeFile(HdfsWriter.java:93)
at --.--.writer.HdfsWriter$$FastClassBySpringCGLIB$$b3eb2ee4.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at --.--.writer.HdfsWriter$$EnhancerBySpringCGLIB$$2e0168e3.writeFile(<generated>)
at --.--.DataFileServiceImpl.ingest(DataFileServiceImpl.java:239)
at --.--.DataFileServiceImpl$$FastClassBySpringCGLIB$$2a815cbc.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at --.--.DataFileServiceImpl$$EnhancerBySpringCGLIB$$21d5ca92.ingest(<generated>)
at --.rest.lake.v1.DataFilesController.upload(DataFilesController.java:307)
at --.rest.lake.v1.DataFilesController$$FastClassBySpringCGLIB$$223d233d.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at --.rest.lake.v1.DataFilesController$$EnhancerBySpringCGLIB$$59a99ea6.upload(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
... 98 common frames omitted

Below is the working code. ie same code logic built different time


2018-10-18 03:28:42.368  INFO 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] alluxio.AbstractClient                    257 : Alluxio client (version 1.8.0) is trying to connect with BlockMasterClient @ <MASTER-HOST>/<MASTER>:31398
2018-10-18 03:28:42.369 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] a.s.a.TransportProviderUtils               47 : Impersonation: subject: Subject:
Principal: --

2018-10-18 03:28:42.369 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] a.s.a.TransportProviderUtils               57 : Impersonation: configured: _--_USER_
2018-10-18 03:28:42.369 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] a.s.a.TransportProviderUtils               73 : Impersonation: --User: -- impersonationUser: --
2018-10-18 03:28:42.370 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       261 : opening transport org.apache.thrift.transport.TSaslClientTransport@41fabf7b
2018-10-18 03:28:42.371 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.t.transport.TSaslClientTransport       96 : Sending mechanism name PLAIN and initial response of length 20
2018-10-18 03:28:42.371 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       162 : CLIENT: Writing message with status START and payload length 5
2018-10-18 03:28:42.371 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       162 : CLIENT: Writing message with status COMPLETE and payload length 20
2018-10-18 03:28:42.372 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       273 : CLIENT: Start message handled
2018-10-18 03:28:42.372 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       296 : CLIENT: Main negotiation loop complete
2018-10-18 03:28:42.372 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       306 : CLIENT: SASL Client receiving last message
2018-10-18 03:28:42.373 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       206 : CLIENT: Received message with status COMPLETE and payload length 0
2018-10-18 03:28:42.373  INFO 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] alluxio.AbstractClient                    264 : Client registered with BlockMasterClient @ <MASTER-HOST>/<MASTER>:31398
2018-10-18 03:28:42.374 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       498 : writing data length: 52
2018-10-18 03:28:42.374 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 45
2018-10-18 03:28:42.382 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       498 : writing data length: 52
2018-10-18 03:28:42.387 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 2199
2018-10-18 03:28:42.477  INFO 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] alluxio.network.netty.NettyChannelPool     93 : Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@76256, remoteAddress: /10.118.249.69:31499).
2018-10-18 03:28:42.621 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       498 : writing data length: 115
2018-10-18 03:28:42.632 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 29
2018-10-18 03:28:42.636 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] u.d.platform.log.MethodLogger              78 : END - --.--.writer.--Writer.writeFile :: END TIME - 1539833322636
2018-10-18 03:28:42.637 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] u.d.platform.log.MethodLogger              64 : START - --.--.reader.--Reader.getContentLength:/--_dev/dev2/test/c0mbqm :: START TIME - 1539833322637
2018-10-18 03:28:42.643 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] alluxio.hadoop.AbstractFileSystem         334 : getFileStatus(alluxio://<MASTER>:31398/--_dev/dev2/test/c0mbqm)
2018-10-18 03:28:42.644 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       498 : writing data length: 108
2018-10-18 03:28:42.645 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] o.a.thrift.transport.TSaslTransport       459 : CLIENT: reading data length: 487
2018-10-18 03:28:42.659 DEBUG 1  --- [e646c208-1198-4bc0-bc3e-ba24c5491920, dev2] [io-14136-exec-1] u.d.platform.log.MethodLogger              78 : END - --.--.reader.--Reader.getContentLength :: END TIM

On Thursday, October 18, 2018 at 9:45:35 PM UTC+5:30, Gene Pang wrote:
Hi Jais,

What was the difference between when it was working and and not working? Were you always on the same 1.8.0 version? Is there more to the stacktrace for "org.springframework.web.util.NestedServletException"? That doesn't look Alluxio related, but maybe there is more info in the stack. Or, is there some Alluxio error anywhere, in the client, master or worker logs?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio netty class load issue

Gene Pang
What is the OS of the host you are running the client and workers on?

Have you tried using this configuration value for the client? alluxio.user.network.netty.channel 

http://www.alluxio.org/docs/1.8/en/reference/Properties-List.html#alluxio.user.network.netty.channel

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.