IN alluxio -map reduce Failed to connect to FileSystemMasterClient @ quickstart.cloudera/172.17.0.2

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

IN alluxio -map reduce Failed to connect to FileSystemMasterClient @ quickstart.cloudera/172.17.0.2

UDIT JOSHI
I passed the map-reduce integration test on cloudera

integration/checker/bin/alluxio-checker.sh mapreduce
integration/checker/bin/alluxio-checker.sh mapreduce SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop-mapreduce/alluxio-1.8.1-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/10/06 18:07:34 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 18/10/06 18:07:40 INFO mapreduce.JobSubmitter: number of splits:10 18/10/06 18:07:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538847821676_0003 18/10/06 18:07:41 INFO impl.YarnClientImpl: Submitted application application_1538847821676_0003 18/10/06 18:07:41 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1538847821676_0003/ 18/10/06 18:07:41 INFO mapreduce.Job: Running job: job_1538847821676_0003 18/10/06 18:07:54 INFO mapreduce.Job: Job job_1538847821676_0003 running in uber mode : false 18/10/06 18:07:54 INFO mapreduce.Job: map 0% reduce 0% 18/10/06 18:08:30 INFO mapreduce.Job: map 10% reduce 0% 18/10/06 18:08:35 INFO mapreduce.Job: map 20% reduce 0% 18/10/06 18:08:36 INFO mapreduce.Job: map 30% reduce 0% 18/10/06 18:08:40 INFO mapreduce.Job: map 40% reduce 0% 18/10/06 18:08:41 INFO mapreduce.Job: map 60% reduce 0% 18/10/06 18:09:05 INFO mapreduce.Job: map 70% reduce 0% 18/10/06 18:09:09 INFO mapreduce.Job: map 90% reduce 0% 18/10/06 18:09:10 INFO mapreduce.Job: map 100% reduce 0% 18/10/06 18:09:11 INFO mapreduce.Job: map 100% reduce 100% 18/10/06 18:09:12 INFO mapreduce.Job: Job job_1538847821676_0003 completed successfully 18/10/06 18:09:12 INFO mapreduce.Job: Counters: 50 File System Counters FILE: Number of bytes read=216 FILE: Number of bytes written=1254343 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=770 HDFS: Number of bytes written=19 HDFS: Number of read operations=23 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Killed map tasks=1 Launched map tasks=11 Launched reduce tasks=1 Other local map tasks=11 Total time spent by all maps in occupied slots (ms)=326727 Total time spent by all reduces in occupied slots (ms)=33689 Total time spent by all map tasks (ms)=326727 Total time spent by all reduce tasks (ms)=33689 Total vcore-seconds taken by all map tasks=326727 Total vcore-seconds taken by all reduce tasks=33689 Total megabyte-seconds taken by all map tasks=334568448 Total megabyte-seconds taken by all reduce tasks=34497536 Map-Reduce Framework Map input records=10 Map output records=10 Map output bytes=190 Map output materialized bytes=270 Input split bytes=770 Combine input records=10 Combine output records=10 Reduce input groups=1 Reduce shuffle bytes=270 Reduce input records=10 Reduce output records=1 Spilled Records=20 Shuffled Maps =10 Failed Shuffles=0 Merged Map outputs=10 GC time elapsed (ms)=5564 CPU time spent (ms)=6370 Physical memory (bytes) snapshot=2825220096 Virtual memory (bytes) snapshot=27831209984 Total committed heap usage (bytes)=2811928576 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=19 ***** The integration checker ran at 06/10/18 18:09:12. ***** Nodes of IP addresses: 172.17.0.2 can recognize Alluxio filesystem. ***** Integration test passed. *****


Running Hadoop wordcount with Alluxio Locally

Start Alluxio locally:

bin/alluxio-start.sh local SudoMount

You can add a sample file to Alluxio to run wordcount on. From your Alluxio directory:

bin/alluxio fs copyFromLocal LICENSE /wordcount/input.txt
 bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /root/alluxio-1.8.1-hadoop-2.6/client/alluxio-1.8.1-client.jar alluxio://172.17.0.2:19998/wordcount/input.txt alluxio://172.17.0.2:19998/wordcount/output
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop-mapreduce/alluxio-1.8.1-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/10/06 18:14:02 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 18:14:02 INFO metrics.MetricsSystem: Starting sinks with config: {}.
18/10/06 18:14:02 INFO file.FileSystemContext: Created filesystem context with id app-1443148504301165618. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 18:14:02 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to bootstrap-connect with quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client has bootstrap-connected with quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:03 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:03 INFO file.FileSystemContext: Created filesystem context with id app-8751183887574023538. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:03 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/10/06 18:14:04 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:04 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 18:14:07 INFO input.FileInputFormat: Total input paths to process : 1
18/10/06 18:14:08 INFO mapreduce.JobSubmitter: number of splits:1
18/10/06 18:14:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538847821676_0004
18/10/06 18:14:09 INFO impl.YarnClientImpl: Submitted application application_1538847821676_0004
18/10/06 18:14:09 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1538847821676_0004/
18/10/06 18:14:09 INFO mapreduce.Job: Running job: job_1538847821676_0004
18/10/06 18:14:22 INFO mapreduce.Job: Job job_1538847821676_0004 running in uber mode : false
18/10/06 18:14:22 INFO mapreduce.Job:  map 0% reduce 0%
18/10/06 18:18:21 INFO mapreduce.Job: Job job_1538847821676_0004 failed with state FAILED due to: Job setup failed : alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ quickstart.cloudera/172.17.0.2:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:322)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:425)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:407)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:106)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:96)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:643)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:27)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1861)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:305)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


18/10/06 18:18:22 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
18/10/06 18:18:23 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: IN alluxio -map reduce Failed to connect to FileSystemMasterClient @ quickstart.cloudera/172.17.0.2

UDIT JOSHI
It shows that error in master.log
Caused by: org.apache.thrift.transport.TTransportException: Plain authentication failed: User yarn is not configured for any impersonation. impersonationUser: root
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 6 more

On Thursday, October 11, 2018 at 11:29:16 AM UTC+5:30, UDIT JOSHI wrote:
I passed the map-reduce integration test on cloudera

integration/checker/bin/alluxio-checker.sh mapreduce
integration/checker/bin/alluxio-checker.sh mapreduce SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop-mapreduce/alluxio-1.8.1-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;">http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/10/06 18:07:34 INFO client.RMProxy: Connecting to ResourceManager at /<a href="http://0.0.0.0:8032" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F0.0.0.0%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEhZniyQDlxwL8SlRJ2OACSREecHA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F0.0.0.0%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEhZniyQDlxwL8SlRJ2OACSREecHA&#39;;return true;">0.0.0.0:8032 18/10/06 18:07:40 INFO mapreduce.JobSubmitter: number of splits:10 18/10/06 18:07:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538847821676_0003 18/10/06 18:07:41 INFO impl.YarnClientImpl: Submitted application application_1538847821676_0003 18/10/06 18:07:41 INFO mapreduce.Job: The url to track the job: <a href="http://quickstart.cloudera:8088/proxy/application_1538847821676_0003/18/10/06" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fquickstart.cloudera%3A8088%2Fproxy%2Fapplication_1538847821676_0003%2F18%2F10%2F06\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF-sxxp7f1BQmY6AsA2CIBgJj1Skw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fquickstart.cloudera%3A8088%2Fproxy%2Fapplication_1538847821676_0003%2F18%2F10%2F06\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF-sxxp7f1BQmY6AsA2CIBgJj1Skw&#39;;return true;">http://quickstart.cloudera:8088/proxy/application_1538847821676_0003/ 18/10/06 18:07:41 INFO mapreduce.Job: Running job: job_1538847821676_0003 18/10/06 18:07:54 INFO mapreduce.Job: Job job_1538847821676_0003 running in uber mode : false 18/10/06 18:07:54 INFO mapreduce.Job: map 0% reduce 0% 18/10/06 18:08:30 INFO mapreduce.Job: map 10% reduce 0% 18/10/06 18:08:35 INFO mapreduce.Job: map 20% reduce 0% 18/10/06 18:08:36 INFO mapreduce.Job: map 30% reduce 0% 18/10/06 18:08:40 INFO mapreduce.Job: map 40% reduce 0% 18/10/06 18:08:41 INFO mapreduce.Job: map 60% reduce 0% 18/10/06 18:09:05 INFO mapreduce.Job: map 70% reduce 0% 18/10/06 18:09:09 INFO mapreduce.Job: map 90% reduce 0% 18/10/06 18:09:10 INFO mapreduce.Job: map 100% reduce 0% 18/10/06 18:09:11 INFO mapreduce.Job: map 100% reduce 100% 18/10/06 18:09:12 INFO mapreduce.Job: Job job_1538847821676_0003 completed successfully 18/10/06 18:09:12 INFO mapreduce.Job: Counters: 50 File System Counters FILE: Number of bytes read=216 FILE: Number of bytes written=1254343 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=770 HDFS: Number of bytes written=19 HDFS: Number of read operations=23 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Killed map tasks=1 Launched map tasks=11 Launched reduce tasks=1 Other local map tasks=11 Total time spent by all maps in occupied slots (ms)=326727 Total time spent by all reduces in occupied slots (ms)=33689 Total time spent by all map tasks (ms)=326727 Total time spent by all reduce tasks (ms)=33689 Total vcore-seconds taken by all map tasks=326727 Total vcore-seconds taken by all reduce tasks=33689 Total megabyte-seconds taken by all map tasks=334568448 Total megabyte-seconds taken by all reduce tasks=34497536 Map-Reduce Framework Map input records=10 Map output records=10 Map output bytes=190 Map output materialized bytes=270 Input split bytes=770 Combine input records=10 Combine output records=10 Reduce input groups=1 Reduce shuffle bytes=270 Reduce input records=10 Reduce output records=1 Spilled Records=20 Shuffled Maps =10 Failed Shuffles=0 Merged Map outputs=10 GC time elapsed (ms)=5564 CPU time spent (ms)=6370 Physical memory (bytes) snapshot=2825220096 Virtual memory (bytes) snapshot=27831209984 Total committed heap usage (bytes)=2811928576 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=19 ***** The integration checker ran at 06/10/18 18:09:12. ***** Nodes of IP addresses: 172.17.0.2 can recognize Alluxio filesystem. ***** Integration test passed. *****


Running Hadoop wordcount with Alluxio Locally

Start Alluxio locally:

bin/alluxio-start.sh local SudoMount

You can add a sample file to Alluxio to run wordcount on. From your Alluxio directory:

bin/alluxio fs copyFromLocal LICENSE /wordcount/input.txt
 bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /root/alluxio-1.8.1-hadoop-2.6/client/alluxio-1.8.1-client.jar alluxio://<a href="http://172.17.0.2:19998/wordcount/input.txt" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG7l5qamlO3joKctE4aMdcVg3KAQA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG7l5qamlO3joKctE4aMdcVg3KAQA&#39;;return true;">172.17.0.2:19998/wordcount/input.txt alluxio://<a href="http://172.17.0.2:19998/wordcount/output" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGbufJhqLBmpeECvd-5eYltb9JVQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGbufJhqLBmpeECvd-5eYltb9JVQ&#39;;return true;">172.17.0.2:19998/wordcount/output
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop-mapreduce/alluxio-1.8.1-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;">http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/10/06 18:14:02 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 18:14:02 INFO metrics.MetricsSystem: Starting sinks with config: {}.
18/10/06 18:14:02 INFO file.FileSystemContext: Created filesystem context with id app-1443148504301165618. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the <a href="http://alluxio.user.app.id" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;">alluxio.user.app.id property
18/10/06 18:14:02 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to bootstrap-connect with quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client has bootstrap-connected with quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:02 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO file.FileSystemContext: Created filesystem context with id app-8751183887574023538. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the <a href="http://alluxio.user.app.id" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;">alluxio.user.app.id property
18/10/06 18:14:03 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:03 INFO client.RMProxy: Connecting to ResourceManager at /<a href="http://0.0.0.0:8032" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F0.0.0.0%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEhZniyQDlxwL8SlRJ2OACSREecHA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F0.0.0.0%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEhZniyQDlxwL8SlRJ2OACSREecHA&#39;;return true;">0.0.0.0:8032
18/10/06 18:14:04 INFO alluxio.AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:04 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998
18/10/06 18:14:07 INFO input.FileInputFormat: Total input paths to process : 1
18/10/06 18:14:08 INFO mapreduce.JobSubmitter: number of splits:1
18/10/06 18:14:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538847821676_0004
18/10/06 18:14:09 INFO impl.YarnClientImpl: Submitted application application_1538847821676_0004
18/10/06 18:14:09 INFO mapreduce.Job: The url to track the job: <a href="http://quickstart.cloudera:8088/proxy/application_1538847821676_0004/18/10/06" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fquickstart.cloudera%3A8088%2Fproxy%2Fapplication_1538847821676_0004%2F18%2F10%2F06\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQiu9h7pQXB8eNFrf-Qu1H4nCXjQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fquickstart.cloudera%3A8088%2Fproxy%2Fapplication_1538847821676_0004%2F18%2F10%2F06\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQiu9h7pQXB8eNFrf-Qu1H4nCXjQ&#39;;return true;">http://quickstart.cloudera:8088/proxy/application_1538847821676_0004/
18/10/06 18:14:09 INFO mapreduce.Job: Running job: job_1538847821676_0004
18/10/06 18:14:22 INFO mapreduce.Job: Job job_1538847821676_0004 running in uber mode : false
18/10/06 18:14:22 INFO mapreduce.Job:  map 0% reduce 0%
18/10/06 18:18:21 INFO mapreduce.Job: Job job_1538847821676_0004 failed with state FAILED due to: Job setup failed : alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ quickstart.cloudera/<a href="http://172.17.0.2:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F172.17.0.2%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmK5j04GjGKDAybx-DDq3zS0Jfag&#39;;return true;">172.17.0.2:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:322)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:425)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:407)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:106)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:96)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:643)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:27)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1861)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:305)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


18/10/06 18:18:22 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
18/10/06 18:18:23 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: IN alluxio -map reduce Failed to connect to FileSystemMasterClient @ quickstart.cloudera/172.17.0.2

Gene Pang
Hi,

If Hadoop is using impersonation, then you will have to enable impersonation on Alluxio as well. You can do this by adding configuration to the Alluxio servers.

Here is the documentation for Alluxio impersonation. http://www.alluxio.org/docs/1.8/en/Security.html#impersonation

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.