alluxio-1.8.0 client problem

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

alluxio-1.8.0 client problem

Reid Chan

I set up an alluxio cluster (version 1.8.0), but when i tried to run spark application on it, spark client showed the following logs:

Server side:
2018-08-16 17:02:04,670 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-08-16 17:02:07,313 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Client side:
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (1) with FileSystemMasterClient @ hadoop1266.bx.momo.com/10.88.78.187:18888: null
18/08/16 17:00:07 INFO AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ hadoop1266.bx.momo.com/10.88.78.187:18888
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (2) with FileSystemMasterClient @ hadoop1266.bx.momo.com/10.88.78.187:18888: null



Looking forward team's reply.

R.C

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Anton Kulaga
I have the same problem when I try to use alluxio initepreter inside Zeppelin. However, it may be a problem of Zeppelin

On Thursday, 16 August 2018 12:05:31 UTC+3, Reid Chan wrote:

I set up an alluxio cluster (version 1.8.0), but when i tried to run spark application on it, spark client showed the following logs:

Server side:
2018-08-16 17:02:04,670 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-08-16 17:02:07,313 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Client side:
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (1) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null
18/08/16 17:00:07 INFO AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (2) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null



Looking forward team's reply.

R.C

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Anton Kulaga
In reply to this post by Reid Chan
Just checked, it was not only a zeppelin problem but also a spark problem. Had the same error when tried to connect with spark 2.3.0

On Thursday, 16 August 2018 12:05:31 UTC+3, Reid Chan wrote:

I set up an alluxio cluster (version 1.8.0), but when i tried to run spark application on it, spark client showed the following logs:

Server side:
2018-08-16 17:02:04,670 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-08-16 17:02:07,313 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Client side:
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (1) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null
18/08/16 17:00:07 INFO AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (2) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null



Looking forward team's reply.

R.C

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Anton Kulaga
I opened an issue on that https://alluxio.atlassian.net/browse/ALLUXIO-3298?atlOrigin=eyJpIjoiMWU3NDk0YjlkMjNiNGY3NGJhMTYzY2E1NDQ3ZDhkZTEiLCJwIjoiaiJ9

On Friday, 17 August 2018 01:52:45 UTC+3, Anton Kulaga wrote:
Just checked, it was not only a zeppelin problem but also a spark problem. Had the same error when tried to connect with spark 2.3.0

On Thursday, 16 August 2018 12:05:31 UTC+3, Reid Chan wrote:

I set up an alluxio cluster (version 1.8.0), but when i tried to run spark application on it, spark client showed the following logs:

Server side:
2018-08-16 17:02:04,670 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-08-16 17:02:07,313 ERROR TThreadPoolServer - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:92)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Client side:
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (1) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null
18/08/16 17:00:07 INFO AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888
18/08/16 17:00:07 WARN AbstractClient: Failed to connect (2) with FileSystemMasterClient @ <a href="http://hadoop1266.bx.momo.com/10.88.78.187:18888" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fhadoop1266.bx.momo.com%2F10.88.78.187%3A18888\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHL9BA1UhzwndBdUYVBuCqYyFib0g&#39;;return true;">hadoop1266.bx.momo.com/10.88.78.187:18888: null



Looking forward team's reply.

R.C

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Calvin Jia
Hi,

Thanks for the detailed reports. I was not able to reproduce this with Spark 2.3.1 and Alluxio 1.8.0.

Could you check your configuration properties for the client connect protocol? This can be a problem when the client is using a different transport than the server (ie. secured vs non-secured or framed vs non-framed).

Note in Alluxio 1.8.0, the client does a bootstrap with the master.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Anton Kulaga
Dear Calvin,

all the configurations are in my docker stack https://github.com/antonkulaga/bigdata-docker/blob/master/bigdata-extended.yml

I tried to follow the default alluxio-spark tutorial and it did not work. In the tutorial they claim that alluxio should work with spark out of the box. I tried a lot of things but I still failed to make spark standalone 2.3.1 work with alluxio 1.8.0
If you were lucky to succeed with this, maybe you can share your config?

I had to swtich off authentification in alluxio as I do not demand any authentification in spark https://github.com/antonkulaga/bigdata-docker/blob/master/conf/alluxio/alluxio-site.properties

alluxio.security.authorization.permission.enabled=false
alluxio.security.authentication.type=NOSASL



On Friday, 17 August 2018 21:07:42 UTC+3, Calvin Jia wrote:
Hi,

Thanks for the detailed reports. I was not able to reproduce this with Spark 2.3.1 and Alluxio 1.8.0.

Could you check your configuration properties for the client connect protocol? This can be a problem when the client is using a different transport than the server (ie. secured vs non-secured or framed vs non-framed).

Note in Alluxio 1.8.0, the client does a <a href="https://github.com/Alluxio/alluxio/blob/v1.8.0/core/common/src/main/java/alluxio/AbstractClient.java#L195" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;">bootstrap with the master.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Calvin Jia
Hi Anton,

You shouldn't need to disable authentication. If you do choose to disable it, you will also need to ensure your Spark application has the same security settings (ie. pass -D in extra options or put alluxio-site.properties in Spark conf folder with the appropriate settings). If you only set the auth type to NOSASL on the server, you will get the exception mentioned earlier in the thread.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Reid Chan
Confirmed, as calvin said.

Solved. Both client and server should have the same authentication type.

My case is to disable both to make spark work.

在 2018年8月21日星期二 UTC+8上午4:00:40,Calvin Jia写道:
Hi Anton,

You shouldn't need to disable authentication. If you do choose to disable it, you will also need to ensure your Spark application has the same security settings (ie. pass -D in extra options or put alluxio-site.properties in Spark conf folder with the appropriate settings). If you only set the auth type to NOSASL on the server, you will get the exception mentioned earlier in the thread.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

binfan
Administrator
Thanks Reid to confirm the workaround 
(i.e., if server sets alluxio.security.authentication.type=NOSASL,
client needs to set alluxio.security.authentication.type=NOSASL too)

Anton, we were able to reproduce this issue locally.
I will try to fix this issue in 1.8.1. The goal is
if server sets alluxio.security.authentication.type=NOSASL,
Alluxio client  does not need to set so again (to match other auth types)

- Bin

On Tuesday, August 21, 2018 at 3:44:37 AM UTC-7, Reid Chan wrote:
Confirmed, as calvin said.

Solved. Both client and server should have the same authentication type.

My case is to disable both to make spark work.

在 2018年8月21日星期二 UTC+8上午4:00:40,Calvin Jia写道:
Hi Anton,

You shouldn't need to disable authentication. If you do choose to disable it, you will also need to ensure your Spark application has the same security settings (ie. pass -D in extra options or put alluxio-site.properties in Spark conf folder with the appropriate settings). If you only set the auth type to NOSASL on the server, you will get the exception mentioned earlier in the thread.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

Reid Chan
In reply to this post by Anton Kulaga
Hi Anton,

Following are what i did.

Client side:
--conf "spark.driver.extraJavaOption=-Dalluxio.user.metrics.collection.enabled=false -Dalluxio.security.authorization.permission.enabled=false -Dalluxio.security.authentication.type=NOSASL"
--conf "spark.executor.extraJavaOptions=-Dalluxio.user.metrics.collection.enabled=false -Dalluxio.security.authorization.permission.enabled=false -Dalluxio.security.authentication.type=NOSASL"

Server side(Master && Worker):
alluxio.security.authorization.permission.enabled=false
alluxio.security.authentication.type=NOSASL

在 2018年8月18日星期六 UTC+8下午11:33:20,Anton Kulaga写道:
Dear Calvin,

all the configurations are in my docker stack <a href="https://github.com/antonkulaga/bigdata-docker/blob/master/bigdata-extended.yml" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fbigdata-extended.yml\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHGe0iZQ3_xyFqGUvCbRACyy6TEng&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fbigdata-extended.yml\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHGe0iZQ3_xyFqGUvCbRACyy6TEng&#39;;return true;">https://github.com/antonkulaga/bigdata-docker/blob/master/bigdata-extended.yml

I tried to follow the default alluxio-spark tutorial and it did not work. In the tutorial they claim that alluxio should work with spark out of the box. I tried a lot of things but I still failed to make spark standalone 2.3.1 work with alluxio 1.8.0
If you were lucky to succeed with this, maybe you can share your config?

I had to swtich off authentification in alluxio as I do not demand any authentification in spark <a href="https://github.com/antonkulaga/bigdata-docker/blob/master/conf/alluxio/alluxio-site.properties" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fconf%2Falluxio%2Falluxio-site.properties\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZj36KopV5DOqkAgdpukATrJM9tg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fconf%2Falluxio%2Falluxio-site.properties\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZj36KopV5DOqkAgdpukATrJM9tg&#39;;return true;">https://github.com/antonkulaga/bigdata-docker/blob/master/conf/alluxio/alluxio-site.properties

alluxio.security.authorization.permission.enabled=false
alluxio.security.authentication.type=NOSASL



On Friday, 17 August 2018 21:07:42 UTC+3, Calvin Jia wrote:
Hi,

Thanks for the detailed reports. I was not able to reproduce this with Spark 2.3.1 and Alluxio 1.8.0.

Could you check your configuration properties for the client connect protocol? This can be a problem when the client is using a different transport than the server (ie. secured vs non-secured or framed vs non-framed).

Note in Alluxio 1.8.0, the client does a <a href="https://github.com/Alluxio/alluxio/blob/v1.8.0/core/common/src/main/java/alluxio/AbstractClient.java#L195" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;">bootstrap with the master.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio-1.8.0 client problem

binfan
Administrator
Hi,

In case you want to patch it by your own,
here is the patch to fix this issue for 1.9-snapshot: https://github.com/Alluxio/alluxio/pull/7795
and the patch to 1.8 branch: https://github.com/Alluxio/alluxio/pull/7797

So hopefully this issue will be solved in the coming 1.8.1 release and later

- Bin

On Wednesday, August 22, 2018 at 1:10:46 AM UTC-7, Reid Chan wrote:
Hi Anton,

Following are what i did.

Client side:
--conf "spark.driver.extraJavaOption=-Dalluxio.user.metrics.collection.enabled=false -Dalluxio.security.authorization.permission.enabled=false -Dalluxio.security.authentication.type=NOSASL"
--conf "spark.executor.extraJavaOptions=-Dalluxio.user.metrics.collection.enabled=false -Dalluxio.security.authorization.permission.enabled=false -Dalluxio.security.authentication.type=NOSASL"

Server side(Master && Worker):
alluxio.security.authorization.permission.enabled=false
alluxio.security.authentication.type=NOSASL

在 2018年8月18日星期六 UTC+8下午11:33:20,Anton Kulaga写道:
Dear Calvin,

all the configurations are in my docker stack <a href="https://github.com/antonkulaga/bigdata-docker/blob/master/bigdata-extended.yml" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fbigdata-extended.yml\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHGe0iZQ3_xyFqGUvCbRACyy6TEng&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fbigdata-extended.yml\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHGe0iZQ3_xyFqGUvCbRACyy6TEng&#39;;return true;">https://github.com/antonkulaga/bigdata-docker/blob/master/bigdata-extended.yml

I tried to follow the default alluxio-spark tutorial and it did not work. In the tutorial they claim that alluxio should work with spark out of the box. I tried a lot of things but I still failed to make spark standalone 2.3.1 work with alluxio 1.8.0
If you were lucky to succeed with this, maybe you can share your config?

I had to swtich off authentification in alluxio as I do not demand any authentification in spark <a href="https://github.com/antonkulaga/bigdata-docker/blob/master/conf/alluxio/alluxio-site.properties" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fconf%2Falluxio%2Falluxio-site.properties\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZj36KopV5DOqkAgdpukATrJM9tg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fantonkulaga%2Fbigdata-docker%2Fblob%2Fmaster%2Fconf%2Falluxio%2Falluxio-site.properties\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZj36KopV5DOqkAgdpukATrJM9tg&#39;;return true;">https://github.com/antonkulaga/bigdata-docker/blob/master/conf/alluxio/alluxio-site.properties

alluxio.security.authorization.permission.enabled=false
alluxio.security.authentication.type=NOSASL



On Friday, 17 August 2018 21:07:42 UTC+3, Calvin Jia wrote:
Hi,

Thanks for the detailed reports. I was not able to reproduce this with Spark 2.3.1 and Alluxio 1.8.0.

Could you check your configuration properties for the client connect protocol? This can be a problem when the client is using a different transport than the server (ie. secured vs non-secured or framed vs non-framed).

Note in Alluxio 1.8.0, the client does a <a href="https://github.com/Alluxio/alluxio/blob/v1.8.0/core/common/src/main/java/alluxio/AbstractClient.java#L195" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Fblob%2Fv1.8.0%2Fcore%2Fcommon%2Fsrc%2Fmain%2Fjava%2Falluxio%2FAbstractClient.java%23L195\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEM2ApL2iyZT2M26n7fXI7oEmqtPA&#39;;return true;">bootstrap with the master.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.