Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

UDIT JOSHI
In alluxio MapReduce 
I distribute alluxio client jar with -libjar
works fine

2018-10-07 11_17_48-abc.tlp - root@192.168.99.101_22 - Bitvise xterm - @hadoop-hive_~_alluxio-1.8.0-.png

 2) Also set 
export HADOOP_CLASSPATH=/root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar:${HADOOP_CLASSPATH}

3) Also distibute it acrross all nodes  by copying it on $HADOOP_HOME/lib





Then why my intregation check failed?

alluxio intregation check.png


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

Lu Qiu
Hi Udit,

Can you re-run your hadoop-mapreduce-examples.jar wordcount command without `-libjars <ALLUXIO_CLIENT_JAR_PATH>` and show us the console output?

Thanks,
Lu

On Sat, Oct 6, 2018 at 10:56 PM UDIT JOSHI <[hidden email]> wrote:
In alluxio MapReduce 
I distribute alluxio client jar with -libjar
works fine

2018-10-07 11_17_48-abc.tlp - root@192.168.99.101_22 - Bitvise xterm - @hadoop-hive_~_alluxio-1.8.0-.png

 2) Also set 
export HADOOP_CLASSPATH=/root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar:${HADOOP_CLASSPATH}

3) Also distibute it acrross all nodes  by copying it on $HADOOP_HOME/lib





Then why my intregation check failed?

alluxio intregation check.png


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

UDIT JOSHI
Yes, I can. 
wordcount.png

On Sun, Oct 7, 2018 at 11:37 AM Lu <[hidden email]> wrote:
Hi Udit,

Can you re-run your hadoop-mapreduce-examples.jar wordcount command without `-libjars <ALLUXIO_CLIENT_JAR_PATH>` and show us the console output?

Thanks,
Lu

On Sat, Oct 6, 2018 at 10:56 PM UDIT JOSHI <[hidden email]> wrote:
In alluxio MapReduce 
I distribute alluxio client jar with -libjar
works fine

2018-10-07 11_17_48-abc.tlp - root@192.168.99.101_22 - Bitvise xterm - @hadoop-hive_~_alluxio-1.8.0-.png

 2) Also set 
export HADOOP_CLASSPATH=/root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar:${HADOOP_CLASSPATH}

3) Also distibute it acrross all nodes  by copying it on $HADOOP_HOME/lib





Then why my intregation check failed?

alluxio intregation check.png


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

Lu Qiu
Hi Udit,

The Alluxio MapReduce integration checker runs the same wordcount example job for twice, the first time without -libjars and the second with -libjars.
You have already passed wordcount example with/without -libjars by your own, so just ignore the checker results.

One possible reason for the first run to fail may be Hadoop needs to be restarted to pick up lib directory jar changes.

Thanks,
Lu


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

2018-10-07 11_17_48-abc.tlp - root@192.168.99.101_22 - Bitvise xterm - @hadoop-hive_~_alluxio-1.8.0-.png (658K) Download Attachment
alluxio intregation check.png (328K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

UDIT JOSHI
Hi Lu,
I restarted my cloudera hadoop services but get the failure result

After that i running hadoop wordcount with Alluxio locally

  bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar alluxio://localhost:19998/wordcount/input.txt alluxio://localhost:19998/wordcount/output


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/10/06 05:56:42 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 05:56:42 INFO metrics.MetricsSystem: Starting sinks with config: {}.
18/10/06 05:56:42 INFO file.FileSystemContext: Created filesystem context with id app-1877985209125271157. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 05:56:42 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 05:56:42 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to bootstrap-connect with quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client has bootstrap-connected with quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO file.FileSystemContext: Created filesystem context with id app-9079992656391005906. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 05:56:44 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/10/06 05:56:45 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:45 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO input.FileInputFormat: Total input paths to process : 1
18/10/06 05:56:47 INFO network.TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=quickstart.cloudera, rack=null)
18/10/06 05:56:47 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with BlockMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO alluxio.AbstractClient: Client registered with BlockMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO mapreduce.JobSubmitter: number of splits:1
18/10/06 05:56:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538805039145_0001
18/10/06 05:56:49 INFO impl.YarnClientImpl: Submitted application application_1538805039145_0001
18/10/06 05:56:49 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1538805039145_0001/
18/10/06 05:56:49 INFO mapreduce.Job: Running job: job_1538805039145_0001
18/10/06 05:57:05 INFO mapreduce.Job: Job job_1538805039145_0001 running in uber mode : false
18/10/06 05:57:05 INFO mapreduce.Job:  map 0% reduce 0%
18/10/06 06:01:05 INFO mapreduce.Job: Job job_1538805039145_0001 failed with state FAILED due to: Job setup failed : alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ localhost/127.0.0.1:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:325)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:428)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:410)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:106)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:96)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:642)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:27)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1861)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:305)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


18/10/06 06:01:05 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
18/10/06 06:01:06 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
[root@quickstart usr]#


What do i do? next







On Tuesday, October 9, 2018 at 1:33:23 AM UTC+5:30, Lu wrote:
Hi Udit,

The Alluxio MapReduce integration checker runs the same wordcount example job for twice, the first time without -libjars and the second with -libjars.
You have already passed wordcount example with/without -libjars by your own, so just ignore the checker results.

One possible reason for the first run to fail may be Hadoop needs to be restarted to pick up lib directory jar changes.

Thanks,
Lu


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Nodes of IP addresses: 172.26.0.5 cannot recognize Alluxio classes.

Lu Qiu
Hi Udit,

Could you share more information about what mode your Alluxio and Hadoop are running on?
Are you running the MapReduce job on YARN in a pseudo-distributed mode?

Thanks,
Lu

On Tue, Oct 9, 2018 at 8:37 PM UDIT JOSHI <[hidden email]> wrote:
Hi Lu,
I restarted my cloudera hadoop services but get the failure result

After that i running hadoop wordcount with Alluxio locally

  bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar alluxio://localhost:19998/wordcount/input.txt alluxio://localhost:19998/wordcount/output


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/alluxio-1.8.0-hadoop-2.6/client/alluxio-1.8.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/10/06 05:56:42 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 05:56:42 INFO metrics.MetricsSystem: Starting sinks with config: {}.
18/10/06 05:56:42 INFO file.FileSystemContext: Created filesystem context with id app-1877985209125271157. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 05:56:42 INFO hadoop.HadoopConfigurationUtils: Loading Alluxio properties from Hadoop configuration: {}
18/10/06 05:56:42 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to bootstrap-connect with quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client has bootstrap-connected with quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ quickstart.cloudera/172.17.0.2:19998
18/10/06 05:56:43 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:43 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO file.FileSystemContext: Created filesystem context with id app-9079992656391005906. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/06 05:56:44 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO alluxio.AbstractClient: Client registered with MetricsMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:44 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/10/06 05:56:45 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:45 INFO alluxio.AbstractClient: Client registered with FileSystemMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO input.FileInputFormat: Total input paths to process : 1
18/10/06 05:56:47 INFO network.TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=quickstart.cloudera, rack=null)
18/10/06 05:56:47 INFO alluxio.AbstractClient: Alluxio client (version 1.8.0) is trying to connect with BlockMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO alluxio.AbstractClient: Client registered with BlockMasterClient @ localhost/127.0.0.1:19998
18/10/06 05:56:47 INFO mapreduce.JobSubmitter: number of splits:1
18/10/06 05:56:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538805039145_0001
18/10/06 05:56:49 INFO impl.YarnClientImpl: Submitted application application_1538805039145_0001
18/10/06 05:56:49 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1538805039145_0001/
18/10/06 05:56:49 INFO mapreduce.Job: Running job: job_1538805039145_0001
18/10/06 05:57:05 INFO mapreduce.Job: Job job_1538805039145_0001 running in uber mode : false
18/10/06 05:57:05 INFO mapreduce.Job:  map 0% reduce 0%
18/10/06 06:01:05 INFO mapreduce.Job: Job job_1538805039145_0001 failed with state FAILED due to: Job setup failed : alluxio.exception.status.UnavailableException: Failed to connect to FileSystemMasterClient @ localhost/127.0.0.1:19998 after 44 attempts
        at alluxio.AbstractClient.connect(AbstractClient.java:325)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:428)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:410)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:106)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:96)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:642)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:27)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1861)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:305)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


18/10/06 06:01:05 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
18/10/06 06:01:06 INFO heartbeat.HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
[root@quickstart usr]#


What do i do? next







On Tuesday, October 9, 2018 at 1:33:23 AM UTC+5:30, Lu wrote:
Hi Udit,

The Alluxio MapReduce integration checker runs the same wordcount example job for twice, the first time without -libjars and the second with -libjars.
You have already passed wordcount example with/without -libjars by your own, so just ignore the checker results.

One possible reason for the first run to fail may be Hadoop needs to be restarted to pick up lib directory jar changes.

Thanks,
Lu


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.