Cannot use alluxio keyvalue store from spark container.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot use alluxio keyvalue store from spark container.

Robin Kuttaiah
Hello,

I need help in using alluxio keyvalue store from streaming spark application.

Env details:
OS -> Oracle Linux Server release 7.4
Spark version -> 2.3.0
Alluxio -> alluxio-1.7.1-bin.tar.gz
Followed link -> https://www.alluxio.org/docs/1.6/en/Running-Alluxio-On-Docker.html

Currently I have spark(yarn), alluxio master and alluxio worker containers. All these containers are in same host.

I have made alluxio-assembly-client-1.7.1-jar-with-dependencies.jar as part of spark classpath.

I have streaming spark application which requires alluxio keyvalue store get/put operations. 
Attached is the java code which does the same and is called with in spark code as below

try {
  String alluxioURI = "alluxio://my.host.com:19998/default_tests_files/robin";
  print(Level.INFO, "Alluxio URI "+alluxioURI);
  AlluxioKeyValueStoreOperations alluxioClient = new AlluxioKeyValueStoreOperations(new AlluxioURI(alluxioURI));
  alluxioClient.put("myKey".getBytes(Charset.forName("UTF-8")), "myValue".getBytes(Charset.forName("UTF-8")));
} catch(Exception exp) {
  print(Level.INFO, "Alluxio write error "+exp.getMessage());
  exp.printStackTrace();
}


When this utility is used with in aluxio master or work container it works without any issue.

But when the same code is called from spark I get below exception. Event though, I give the same URI, i.e "alluxio://my.host.com:19998/default_tests_files/robin", it somehow picks "yarn-nm/172.18.0.8:19998".  yarn-nm is the spark/kafka container what I have.

Below is the exception I get:

18/08/02 15:37:02 INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (0) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (1) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (2) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (3) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (4) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (5) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.Con
18/08/02 15:41:01 WARN AbstractClient: Failed to connect (43) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 WARN AbstractClient: Failed to connect (44) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java:71)

It looks like alluxio master and worker container can see each other but the spark container what I have cannot see master/worker.

How this issue can be solved? 

Any help would be highly appreciated.

regards,
Robin Kuttaiah

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

AlluxioKeyValueStoreOperations.java (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Cannot use alluxio keyvalue store from spark container.

Andrew Audibert
Sorry, the native API is pretty misleading at the moment. It doesn't actually use the hostname/port portion of the AlluxioURI - it only uses the path. The hostname and port are taken from configuration properties. To get the client to connect to the my.host.com, you need to set the "alluxio.master.hostname" property to "my.host.com". This can be done in code with
Configuration.set(alluxio.PropertyKey.MASTER_HOSTNAME, "my.host.com");

There are other ways to set configuration properties, such as putting an alluxio-site.properties file on your classpath, or passing JVM properties to the java process. See https://www.alluxio.org/docs/1.8/en/Configuration-Settings.html for more details.

Hope that helps,
Andrew

On Thu, Aug 2, 2018 at 11:02 AM Robin Kuttaiah <[hidden email]> wrote:
Hello,

I need help in using alluxio keyvalue store from streaming spark application.

Env details:
OS -> Oracle Linux Server release 7.4
Spark version -> 2.3.0
Alluxio -> alluxio-1.7.1-bin.tar.gz

Currently I have spark(yarn), alluxio master and alluxio worker containers. All these containers are in same host.

I have made alluxio-assembly-client-1.7.1-jar-with-dependencies.jar as part of spark classpath.

I have streaming spark application which requires alluxio keyvalue store get/put operations. 
Attached is the java code which does the same and is called with in spark code as below

try {
  String alluxioURI = "alluxio://my.host.com:19998/default_tests_files/robin";
  print(Level.INFO, "Alluxio URI "+alluxioURI);
  AlluxioKeyValueStoreOperations alluxioClient = new AlluxioKeyValueStoreOperations(new AlluxioURI(alluxioURI));
  alluxioClient.put("myKey".getBytes(Charset.forName("UTF-8")), "myValue".getBytes(Charset.forName("UTF-8")));
} catch(Exception exp) {
  print(Level.INFO, "Alluxio write error "+exp.getMessage());
  exp.printStackTrace();
}


When this utility is used with in aluxio master or work container it works without any issue.

But when the same code is called from spark I get below exception. Event though, I give the same URI, i.e "alluxio://my.host.com:19998/default_tests_files/robin", it somehow picks "yarn-nm/172.18.0.8:19998".  yarn-nm is the spark/kafka container what I have.

Below is the exception I get:

18/08/02 15:37:02 INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (0) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (1) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (2) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (3) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (4) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (5) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.Con
18/08/02 15:41:01 WARN AbstractClient: Failed to connect (43) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 WARN AbstractClient: Failed to connect (44) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java:71)

It looks like alluxio master and worker container can see each other but the spark container what I have cannot see master/worker.

How this issue can be solved? 

Any help would be highly appreciated.

regards,
Robin Kuttaiah

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot use alluxio keyvalue store from spark container.

Robin Kuttaiah
Hi Andrew,

Thanks for the support.
With mentioned changes I get below;

Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
alluxio.exception.AccessControlException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
Caused by: alluxio.exception.status.PermissionDeniedException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x


since spark runs on yarn, it looks yarn user insight does not have privileges hence am using hdfs location..

I recreated the container with hdfs path as below;


docker run -d --net=host \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio master
             
             
export INSTANCE_PUBLIC_IP=10.10.10.10
docker run -d --net=host \
             -v /mnt/ramdisk:/opt/ramdisk \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_MASTER_HOSTNAME=${INSTANCE_PUBLIC_IP} \
             -e ALLUXIO_RAM_FOLDER=/opt/ramdisk \
             -e ALLUXIO_WORKER_MEMORY_SIZE=1GB \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio worker

With this it looks like spark executor which executes on the spark worker node fails to connect.  I see, its pinging to correct IP but still fails.

INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java


Any pointers?

regards,
Robin Kuttaiah.



On Thu, Aug 2, 2018 at 11:58 PM, Andrew Audibert <[hidden email]> wrote:
Sorry, the native API is pretty misleading at the moment. It doesn't actually use the hostname/port portion of the AlluxioURI - it only uses the path. The hostname and port are taken from configuration properties. To get the client to connect to the my.host.com, you need to set the "alluxio.master.hostname" property to "my.host.com". This can be done in code with
Configuration.set(alluxio.PropertyKey.MASTER_HOSTNAME, "my.host.com");

There are other ways to set configuration properties, such as putting an alluxio-site.properties file on your classpath, or passing JVM properties to the java process. See https://www.alluxio.org/docs/1.8/en/Configuration-Settings.html for more details.

Hope that helps,
Andrew

On Thu, Aug 2, 2018 at 11:02 AM Robin Kuttaiah <[hidden email]> wrote:
Hello,

I need help in using alluxio keyvalue store from streaming spark application.

Env details:
OS -> Oracle Linux Server release 7.4
Spark version -> 2.3.0
Alluxio -> alluxio-1.7.1-bin.tar.gz

Currently I have spark(yarn), alluxio master and alluxio worker containers. All these containers are in same host.

I have made alluxio-assembly-client-1.7.1-jar-with-dependencies.jar as part of spark classpath.

I have streaming spark application which requires alluxio keyvalue store get/put operations. 
Attached is the java code which does the same and is called with in spark code as below

try {
  String alluxioURI = "alluxio://my.host.com:19998/default_tests_files/robin";
  print(Level.INFO, "Alluxio URI "+alluxioURI);
  AlluxioKeyValueStoreOperations alluxioClient = new AlluxioKeyValueStoreOperations(new AlluxioURI(alluxioURI));
  alluxioClient.put("myKey".getBytes(Charset.forName("UTF-8")), "myValue".getBytes(Charset.forName("UTF-8")));
} catch(Exception exp) {
  print(Level.INFO, "Alluxio write error "+exp.getMessage());
  exp.printStackTrace();
}


When this utility is used with in aluxio master or work container it works without any issue.

But when the same code is called from spark I get below exception. Event though, I give the same URI, i.e "alluxio://my.host.com:19998/default_tests_files/robin", it somehow picks "yarn-nm/172.18.0.8:19998".  yarn-nm is the spark/kafka container what I have.

Below is the exception I get:

18/08/02 15:37:02 INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (0) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (1) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (2) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (3) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (4) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (5) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.Con
18/08/02 15:41:01 WARN AbstractClient: Failed to connect (43) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 WARN AbstractClient: Failed to connect (44) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java:71)

It looks like alluxio master and worker container can see each other but the spark container what I have cannot see master/worker.

How this issue can be solved? 

Any help would be highly appreciated.

regards,
Robin Kuttaiah

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot use alluxio keyvalue store from spark container.

Andrew Audibert
If the address is correct, the issue might be with the master. Are you able to access the master from other clients like the CLI or web UI? Also make sure the master process is still running and check the logs at logs/master.log.

- Andrew

On Fri, Aug 3, 2018 at 2:02 AM Kuttaiah Robin <[hidden email]> wrote:
Hi Andrew,

Thanks for the support.
With mentioned changes I get below;

Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
alluxio.exception.AccessControlException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
Caused by: alluxio.exception.status.PermissionDeniedException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x


since spark runs on yarn, it looks yarn user insight does not have privileges hence am using hdfs location..

I recreated the container with hdfs path as below;


docker run -d --net=host \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio master
             
             
export INSTANCE_PUBLIC_IP=10.10.10.10
docker run -d --net=host \
             -v /mnt/ramdisk:/opt/ramdisk \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_MASTER_HOSTNAME=${INSTANCE_PUBLIC_IP} \
             -e ALLUXIO_RAM_FOLDER=/opt/ramdisk \
             -e ALLUXIO_WORKER_MEMORY_SIZE=1GB \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio worker

With this it looks like spark executor which executes on the spark worker node fails to connect.  I see, its pinging to correct IP but still fails.

INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java


Any pointers?

regards,
Robin Kuttaiah.



On Thu, Aug 2, 2018 at 11:58 PM, Andrew Audibert <[hidden email]> wrote:
Sorry, the native API is pretty misleading at the moment. It doesn't actually use the hostname/port portion of the AlluxioURI - it only uses the path. The hostname and port are taken from configuration properties. To get the client to connect to the my.host.com, you need to set the "alluxio.master.hostname" property to "my.host.com". This can be done in code with
Configuration.set(alluxio.PropertyKey.MASTER_HOSTNAME, "my.host.com");

There are other ways to set configuration properties, such as putting an alluxio-site.properties file on your classpath, or passing JVM properties to the java process. See https://www.alluxio.org/docs/1.8/en/Configuration-Settings.html for more details.

Hope that helps,
Andrew

On Thu, Aug 2, 2018 at 11:02 AM Robin Kuttaiah <[hidden email]> wrote:
Hello,

I need help in using alluxio keyvalue store from streaming spark application.

Env details:
OS -> Oracle Linux Server release 7.4
Spark version -> 2.3.0
Alluxio -> alluxio-1.7.1-bin.tar.gz

Currently I have spark(yarn), alluxio master and alluxio worker containers. All these containers are in same host.

I have made alluxio-assembly-client-1.7.1-jar-with-dependencies.jar as part of spark classpath.

I have streaming spark application which requires alluxio keyvalue store get/put operations. 
Attached is the java code which does the same and is called with in spark code as below

try {
  String alluxioURI = "alluxio://my.host.com:19998/default_tests_files/robin";
  print(Level.INFO, "Alluxio URI "+alluxioURI);
  AlluxioKeyValueStoreOperations alluxioClient = new AlluxioKeyValueStoreOperations(new AlluxioURI(alluxioURI));
  alluxioClient.put("myKey".getBytes(Charset.forName("UTF-8")), "myValue".getBytes(Charset.forName("UTF-8")));
} catch(Exception exp) {
  print(Level.INFO, "Alluxio write error "+exp.getMessage());
  exp.printStackTrace();
}


When this utility is used with in aluxio master or work container it works without any issue.

But when the same code is called from spark I get below exception. Event though, I give the same URI, i.e "alluxio://my.host.com:19998/default_tests_files/robin", it somehow picks "yarn-nm/172.18.0.8:19998".  yarn-nm is the spark/kafka container what I have.

Below is the exception I get:

18/08/02 15:37:02 INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (0) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (1) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (2) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (3) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (4) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (5) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.Con
18/08/02 15:41:01 WARN AbstractClient: Failed to connect (43) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 WARN AbstractClient: Failed to connect (44) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java:71)

It looks like alluxio master and worker container can see each other but the spark container what I have cannot see master/worker.

How this issue can be solved? 

Any help would be highly appreciated.

regards,
Robin Kuttaiah

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot use alluxio keyvalue store from spark container.

Robin Kuttaiah
Hi Andrew,

It works now.

I had hdfs container created with user1 and alluxio container with user2.
Hence alluxio cannot write into hdfs because of permission issue and is expected.

There are two ways we can solve:
1. Add required user into hdfs container.
2. Create alluxio container with same user as hdfs.

I did option 2 and it works fine for me. 

Thanks for the support as usual.

On Fri, Aug 3, 2018 at 10:18 PM, Andrew Audibert <[hidden email]> wrote:
If the address is correct, the issue might be with the master. Are you able to access the master from other clients like the CLI or web UI? Also make sure the master process is still running and check the logs at logs/master.log.

- Andrew

On Fri, Aug 3, 2018 at 2:02 AM Kuttaiah Robin <[hidden email]> wrote:
Hi Andrew,

Thanks for the support.
With mentioned changes I get below;

Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
alluxio.exception.AccessControlException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x
Caused by: alluxio.exception.status.PermissionDeniedException: Permission denied: user=insight, access=-w-, path=/default_tests_files/robin: failed at default_tests_files, inode owner=root, inode group=root, inode mode=rwxr-xr-x


since spark runs on yarn, it looks yarn user insight does not have privileges hence am using hdfs location..

I recreated the container with hdfs path as below;


docker run -d --net=host \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio master
             
             
export INSTANCE_PUBLIC_IP=10.10.10.10
docker run -d --net=host \
             -v /mnt/ramdisk:/opt/ramdisk \
             -v $PWD/underStorage:/underStorage \
             -e ALLUXIO_MASTER_HOSTNAME=${INSTANCE_PUBLIC_IP} \
             -e ALLUXIO_RAM_FOLDER=/opt/ramdisk \
             -e ALLUXIO_WORKER_MEMORY_SIZE=1GB \
             -e ALLUXIO_UNDERFS_ADDRESS=hdfs://my.host.com:9000/insight \
             alluxio worker

With this it looks like spark executor which executes on the spark worker node fails to connect.  I see, its pinging to correct IP but still fails.

INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @10.10.10.10/10.10.10.10 :19998 after 43 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java


Any pointers?

regards,
Robin Kuttaiah.



On Thu, Aug 2, 2018 at 11:58 PM, Andrew Audibert <[hidden email]> wrote:
Sorry, the native API is pretty misleading at the moment. It doesn't actually use the hostname/port portion of the AlluxioURI - it only uses the path. The hostname and port are taken from configuration properties. To get the client to connect to the my.host.com, you need to set the "alluxio.master.hostname" property to "my.host.com". This can be done in code with
Configuration.set(alluxio.PropertyKey.MASTER_HOSTNAME, "my.host.com");

There are other ways to set configuration properties, such as putting an alluxio-site.properties file on your classpath, or passing JVM properties to the java process. See https://www.alluxio.org/docs/1.8/en/Configuration-Settings.html for more details.

Hope that helps,
Andrew

On Thu, Aug 2, 2018 at 11:02 AM Robin Kuttaiah <[hidden email]> wrote:
Hello,

I need help in using alluxio keyvalue store from streaming spark application.

Env details:
OS -> Oracle Linux Server release 7.4
Spark version -> 2.3.0
Alluxio -> alluxio-1.7.1-bin.tar.gz

Currently I have spark(yarn), alluxio master and alluxio worker containers. All these containers are in same host.

I have made alluxio-assembly-client-1.7.1-jar-with-dependencies.jar as part of spark classpath.

I have streaming spark application which requires alluxio keyvalue store get/put operations. 
Attached is the java code which does the same and is called with in spark code as below

try {
  String alluxioURI = "alluxio://my.host.com:19998/default_tests_files/robin";
  print(Level.INFO, "Alluxio URI "+alluxioURI);
  AlluxioKeyValueStoreOperations alluxioClient = new AlluxioKeyValueStoreOperations(new AlluxioURI(alluxioURI));
  alluxioClient.put("myKey".getBytes(Charset.forName("UTF-8")), "myValue".getBytes(Charset.forName("UTF-8")));
} catch(Exception exp) {
  print(Level.INFO, "Alluxio write error "+exp.getMessage());
  exp.printStackTrace();
}


When this utility is used with in aluxio master or work container it works without any issue.

But when the same code is called from spark I get below exception. Event though, I give the same URI, i.e "alluxio://my.host.com:19998/default_tests_files/robin", it somehow picks "yarn-nm/172.18.0.8:19998".  yarn-nm is the spark/kafka container what I have.

Below is the exception I get:

18/08/02 15:37:02 INFO AlluxioKeyValueStoreOperations: Putting key-value pairs...
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (0) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (1) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (2) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:02 WARN AbstractClient: Failed to connect (3) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (4) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:37:03 WARN AbstractClient: Failed to connect (5) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.Con
18/08/02 15:41:01 WARN AbstractClient: Failed to connect (43) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 WARN AbstractClient: Failed to connect (44) with KeyValueMasterClient @ yarn-nm/172.18.0.8:19998: java.net.ConnectException: Connection refused
18/08/02 15:41:02 INFO HDFSUtil: Alluxio write error Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
alluxio.exception.status.UnavailableException: Failed to connect to KeyValueMasterClient @ yarn-nm/172.18.0.8:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:230)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:302)
        at alluxio.client.keyvalue.KeyValueMasterClient.createStore(KeyValueMasterClient.java:111)
        at alluxio.client.keyvalue.BaseKeyValueStoreWriter.<init>(BaseKeyValueStoreWriter.java:71)

It looks like alluxio master and worker container can see each other but the spark container what I have cannot see master/worker.

How this issue can be solved? 

Any help would be highly appreciated.

regards,
Robin Kuttaiah

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.