Re: Tez over Tachyon

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Tez over Tachyon

ashish kumar
Hi All,

I started using Hive with Alluxio. And after few initial hurdles I'm able to make it work. Now, Hive with MR is working fine but the same query failing with Hive on TEZ engine.

Hive Version 1.2
Alluxio Version 1.0.1
TEZ version 0.7

Below is the error log:

Error:

org.apache.tez.dag.api.TezUncheckedException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:80)
	at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:98)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:137)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:114)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4392)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.access$4300(VertexImpl.java:199)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:3233)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3183)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3164)
	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1812)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:198)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1943)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1929)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:69)
	... 20 more
Caused by: java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:419)
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:308)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:92)
	... 25 more
2016-08-11 08:37:57,399 ERROR [Dispatcher thread: Central] impl.VertexImpl: Can't handle Invalid event V_START on vertex Map 1 with vertexId vertex_1470806535809_0006_1_00 at current state NEW
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: V_START at NEW
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1812)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:198)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1943)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1929)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
	at java.lang.Thread.run(Thread.java:745)

Anybody have any idea?

-Ashish

On Thursday, 3 December 2015 21:25:09 UTC+5:30, Jiří Šimša wrote:
Hi Nishant,

This looks like a configuration error outside of Tachyon and it is not clear to me what is causing it. Is that the full stack trace? Have you tried running the same command with an HDFS URI instead of Tachyon URI?

Best,

--
Jiří Šimša

On Mon, Nov 30, 2015 at 8:31 AM, Nishant Agrawal <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">agraw...@...> wrote:
Hi Jiri,

Sorry for the confusion. The actually error stack trace is pasted below. I mistakenly pasted wrong error.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;">http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-29 23:44:36,217 INFO  [main]  (AbstractTFS.java:initialize(331)) - initialize(tachyon://<a href="http://10.10.1.1:19998/wordcount/input.txt" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;">10.10.1.1:19998/wordcount/input.txt, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Tachyon: tachyon://<a href="http://10.10.1.1:19998/wordcount/input.txt" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;">10.10.1.1:19998/wordcount/input.txt
2015-11-29 23:44:36,222 INFO  [main]  (ConfUtils.java:loadFromHadoopConfiguration(91)) - Loading Tachyon properties from Hadoop configuration: {}
2015-11-29 23:44:36,241 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with BlockMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 23:44:36,251 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with BlockMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 23:44:36,272 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with FileSystemMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 23:44:36,273 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with FileSystemMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 23:44:36,280 INFO  [main]  (AbstractTFS.java:initialize(352)) - tachyon://<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998 tachyon://<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998 hdfs://<a href="http://10.10.1.1:8020" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4qzgzzhy2g58FWbePBDLHp83mVg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4qzgzzhy2g58FWbePBDLHp83mVg&#39;;return true;">10.10.1.1:8020
2015-11-29 23:44:36,280 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 23:44:36,281 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 23:44:37,238 INFO  [main] impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: <a href="http://10.10.1.1:8188/ws/v1/timeline/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8188%2Fws%2Fv1%2Ftimeline%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFE0uPHW_ClPW5jDDiOgOftuPvCtA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8188%2Fws%2Fv1%2Ftimeline%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFE0uPHW_ClPW5jDDiOgOftuPvCtA&#39;;return true;">http://10.10.1.1:8188/ws/v1/timeline/
2015-11-29 23:44:37,367 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /<a href="http://10.10.1.1:8032" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNbfv9hb3cghOovjmWIk7GiPH16w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNbfv9hb3cghOovjmWIk7GiPH16w&#39;;return true;">10.10.1.1:8032
2015-11-29 23:44:37,492 INFO  [main] client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at /<a href="http://10.10.1.1:10200" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A10200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHC_PfZu9y1d6vIH3tuBK7E7Nz_A&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A10200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHC_PfZu9y1d6vIH3tuBK7E7Nz_A&#39;;return true;">10.10.1.1:10200
2015-11-29 23:44:37,536 INFO  [main] mapreduce.Cluster (Cluster.java:initialize(113)) - Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
java.io.IOException: Cannot initialize Cluster. Please check your configuration for <a href="http://mapreduce.framework.name" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fmapreduce.framework.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFXKGHNDgh7HaE2cpjDWu-FdXlWrA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fmapreduce.framework.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFXKGHNDgh7HaE2cpjDWu-FdXlWrA&#39;;return true;">mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


Regards,
Nishant


On Mon, Nov 30, 2015 at 9:05 AM, Jiří Šimša <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jiri....@...> wrote:
Hi Nishant,

I don't think that the error you are seeing is related to core-site.xml settings. The issue is that you are trying create a file that already exists in Tachyon. In that case, your job will fail because Tachyon does not allow your job to overwrite an existing file. This could for instance happen if you try to run the same word count example (with the same input / output arguments) multiple times. To run the same word count example again, you first need to remove the "output" file created by the previous run of the job. To do that, you can use the following command: "${TACHYON_HOME}/bin/tachyon tfs rmr /wordcount/output".

Hope this helps.

Best,

--
Jiří Šimša

On Sun, Nov 29, 2015 at 10:16 PM, Nishant Agrawal <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">agraw...@...> wrote:
Hi Jiri,

I wasn't able to run Map/Reduce on top of Tez. I was able to run just pure MapReduce jobs.

Current stack look like:
Hive
MR
Tachyon
HDFS

In this stack if I just run the MR jobs it runs perfectly. I just followed the steps given on Tachyon website. However, if I try to run any Hive query it fails.

To make Hive read/write to Tachyon, I have modified core-site.xml of hadoop to make Tachyon as default dfs. But it's causing below error:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHviJfiUvjiSFk2iaVebG-XCN1w-Q&#39;;return true;">http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-29 22:52:47,882 INFO  [main]  (AbstractTFS.java:initialize(331)) - initialize(tachyon://<a href="http://10.10.1.1:19998/wordcount/input.txt" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;">10.10.1.1:19998/wordcount/input.txt, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Tachyon: tachyon://<a href="http://10.10.1.1:19998/wordcount/input.txt" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Finput.txt\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFl4xCF1jjrMTaH8jWAiXkPhkBgxQ&#39;;return true;">10.10.1.1:19998/wordcount/input.txt
2015-11-29 22:52:47,893 INFO  [main]  (ConfUtils.java:loadFromHadoopConfiguration(91)) - Loading Tachyon properties from Hadoop configuration: {tachyon.user.file.understoragetype.default=SYNC_PERSIST}
2015-11-29 22:52:47,939 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with BlockMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 22:52:47,962 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with BlockMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 22:52:48,013 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with FileSystemMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 22:52:48,014 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with FileSystemMaster master @ /<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998
2015-11-29 22:52:48,030 INFO  [main]  (AbstractTFS.java:initialize(352)) - tachyon://<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998 tachyon://<a href="http://10.10.1.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq9hEtC_hlDPuU2vKrXm0UEwXIJA&#39;;return true;">10.10.1.1:19998 hdfs://<a href="http://10.10.1.1:8020" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4qzgzzhy2g58FWbePBDLHp83mVg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4qzgzzhy2g58FWbePBDLHp83mVg&#39;;return true;">10.10.1.1:8020
2015-11-29 22:52:48,031 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 22:52:48,031 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 22:52:48,881 INFO  [main] impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: <a href="http://10.10.1.1:8188/ws/v1/timeline/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8188%2Fws%2Fv1%2Ftimeline%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFE0uPHW_ClPW5jDDiOgOftuPvCtA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8188%2Fws%2Fv1%2Ftimeline%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFE0uPHW_ClPW5jDDiOgOftuPvCtA&#39;;return true;">http://10.10.1.1:8188/ws/v1/timeline/
2015-11-29 22:52:49,011 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /<a href="http://10.10.1.1:8032" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNbfv9hb3cghOovjmWIk7GiPH16w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8032\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFNbfv9hb3cghOovjmWIk7GiPH16w&#39;;return true;">10.10.1.1:8032
2015-11-29 22:52:49,137 INFO  [main] client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at /<a href="http://10.10.1.1:10200" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A10200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHC_PfZu9y1d6vIH3tuBK7E7Nz_A&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A10200\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHC_PfZu9y1d6vIH3tuBK7E7Nz_A&#39;;return true;">10.10.1.1:10200
2015-11-29 22:52:49,525 INFO  [main]  (AbstractTFS.java:getFileStatus(271)) - getFileStatus(tachyon://<a href="http://10.10.1.1:19998/wordcount/output" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;">10.10.1.1:19998/wordcount/output): HDFS Path: hdfs://<a href="http://10.10.1.1:8020/wordcount/output" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHqksxM2aCpTNOb3AecYXLyv_FzQQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A8020%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHqksxM2aCpTNOb3AecYXLyv_FzQQ&#39;;return true;">10.10.1.1:8020/wordcount/output TPath: tachyon://<a href="http://10.10.1.1:19998/wordcount/output" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;">10.10.1.1:19998/wordcount/output
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory tachyon://<a href="http://10.10.1.1:19998/wordcount/output" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F10.10.1.1%3A19998%2Fwordcount%2Foutput\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHK9I6_48cykNmZYQp-dMBWt8BFMg&#39;;return true;">10.10.1.1:19998/wordcount/output already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


With the change in core-site.xml even MR jobs are failing. To run the MR jobs successfully over Tachyon I had to give hdfs as default dfs in core-site.xml. I think, MR jobs reads from Tachyon because it gets hint from I/P & O/P files. But for hive I don't see any other option than changing core-site.xml ( which apparently isn't working ).

Any help to run Hive over MR and Tachyon is greatly appreciated.

Regards,
Nishant

Regards,
Nishant


On Sat, Nov 28, 2015 at 4:33 AM, Jiří Šimša <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jiri....@...> wrote:
Hi Nishant,

Could you please let the mailing list know the steps you followed to run a MapReduce job on top of Tez + Tachyon? As for Hive, could you describe your setup and the problem you are experiencing in more detail? Thanks.

Best,

--
Jiří Šimša

On Wed, Nov 25, 2015 at 6:41 PM, NISHANT AGRAWAL <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">nish191a...@...> wrote:
Hi Gene,

We are able to run the MapReduce jobs on top of tachyon but we are not able to run Hive jobs using MR as execution framework.
Do you have any idea if hive is supported with Tachyon or not? Any pointers will be of good help.

Regards,
Nishant

On Tuesday, 10 November 2015 08:55:27 UTC-6, Gene Pang wrote:
Hi Nishant,

Thanks for your interest in running Tez on Tachyon!

I am not aware of others running Tez on Tachyon, so I don't think there is a guide for running Tez on Tachyon. However, since Tez is a distributed framework to run DAGs for processing data (sounds similar to Spark?), Tachyon should be able to work with Tez. Feel free to ask additional questions as you get into deploying Tez on Tachyon.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">tachyon-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Tachyon Users" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/tachyon-users/-fBVsEAJmuM/unsubscribe" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/topic/tachyon-users/-fBVsEAJmuM/unsubscribe&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/topic/tachyon-users/-fBVsEAJmuM/unsubscribe&#39;;return true;">https://groups.google.com/d/topic/tachyon-users/-fBVsEAJmuM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciYssb2OCwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">tachyon-user...@googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.




--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Tez over Tachyon

Gene Pang
Hi,

I am not familiar with how Tez works, but does Tez use Hive? Does Hive use Tez? In the errors you have shown, I don't see any Alluxio errors in there?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Tez over Tachyon

ashish kumar
Hi Gene,

Hive is having TEZ as a processing engine like MR. I was just expecting if there is any compatible version of TEZ can be used with Alluxio. Because the error shows some dependency versions mismatch.

Thanks,
Ashish

On Thursday, 11 August 2016 19:24:50 UTC+5:30, Gene Pang wrote:
Hi,

I am not familiar with how Tez works, but does Tez use Hive? Does Hive use Tez? In the errors you have shown, I don't see any Alluxio errors in there?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Tez over Tachyon

Gene Pang
Hi Ashish,

I see. It is possible that there are some dependency conflicts. Is there an error message that shows that?

One thing you could try is to build Alluxio with the same hadoop version that Tez is compiled with. Details can be found here: http://www.alluxio.org/docs/master/en/Building-Alluxio-Master-Branch.html#distro-support

It would also be helpful to know which package is in conflict.

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Tez over Tachyon

Gene Pang
Hi Ashish,

Were you able to resolve your issue?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.