EMR 5.6 and Spark 2.1.1

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

EMR 5.6 and Spark 2.1.1

Meenambigai Manivel
I have standalone mode of alluxio installed alluxio-1.7.1-hadoop-2.7-bin.tar.gz on master node
Save a simple text file through the alluxio fs command . 
launched spark shell passing in client jar bundled along with the package.
Reading the textfile from the alluxio is failing 

scala> val s = sc.textFile("alluxio://localhost:19998/LDS_PROPERTIES")
s: org.apache.spark.rdd.RDD[String] = alluxio://localhost:19998/LDS_PROPERTIES MapPartitionsRDD[1] at textFile at <console>:24

scala> s.map(line => println)
res0: org.apache.spark.rdd.RDD[Unit] = MapPartitionsRDD[2] at map at <console>:27

scala> s.map(line => println).collect
[Stage 0:>                                                          (0 + 2) / 2]18/04/25 17:25:47 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, ip-10-237-191-223.aws-w-np.nielsencsp.net, executor 1): alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ localhost/127.0.0.1:19998 after 44 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:230)
at alluxio.hadoop.AbstractFileSystem.initializeInternal(AbstractFileSystem.java:518)

Please advice
 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: EMR 5.6 and Spark 2.1.1

Andrew Audibert
Hi Meenambigai,

The error indicates that the alluxio master is not running on the localhost of the spark executor. To fix this, replace "localhost" with the address of the alluxio master (alluxio://<<<alluxio_master_hostname>>>:19998/LDS_PROPERTIES)

- Andrew

On Wed, Apr 25, 2018 at 10:28 AM Meenambigai Manivel <[hidden email]> wrote:
I have standalone mode of alluxio installed alluxio-1.7.1-hadoop-2.7-bin.tar.gz on master node
Save a simple text file through the alluxio fs command . 
launched spark shell passing in client jar bundled along with the package.
Reading the textfile from the alluxio is failing 

scala> val s = sc.textFile("alluxio://localhost:19998/LDS_PROPERTIES")
s: org.apache.spark.rdd.RDD[String] = alluxio://localhost:19998/LDS_PROPERTIES MapPartitionsRDD[1] at textFile at <console>:24

scala> s.map(line => println)
res0: org.apache.spark.rdd.RDD[Unit] = MapPartitionsRDD[2] at map at <console>:27

scala> s.map(line => println).collect
[Stage 0:>                                                          (0 + 2) / 2]18/04/25 17:25:47 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, ip-10-237-191-223.aws-w-np.nielsencsp.net, executor 1): alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ localhost/127.0.0.1:19998 after 44 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:230)
at alluxio.hadoop.AbstractFileSystem.initializeInternal(AbstractFileSystem.java:518)

Please advice
 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.