Alluxio with Hive on EMR

classic Classic list List threaded Threaded
11 messages Options
Dan
Reply | Threaded
Open this post in threaded view
|

Alluxio with Hive on EMR

Dan
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Boe H
Any luck?  Would be great to hear of successful deployment on EMR.

On Friday, January 20, 2017 at 3:08:10 PM UTC-5, Dan wrote:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

TeddyBear1314
In reply to this post by Dan
Can you provide all the properties in hive-site.xml?Because you init hive with dbType = derby but it complaints that it's not a MYSQL syntax error.
And the Tez error seems that your HDFS URI is incomplete(I think it should be hdfs://host:port, not hdfs:///).



在 2017年1月21日星期六 UTC+8上午4:08:10,Dan写道:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

TeddyBear1314
In reply to this post by Dan

Can you provide all the properties in hive-site.xml?Because you init hive with dbType = derby but it complaints that it's not a MYSQL syntax.And the Tez error seems that your HDFS URI is incomplete(I think it should be hdfs://host:port, not hdfs:///).

在 2017年1月21日星期六 UTC+8上午4:08:10,Dan写道:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Dan
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Dan
Sure.

hive-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Licensed to the Apache Software Foundation (ASF) under one or more       -->
<!-- contributor license agreements.  See the NOTICE file distributed with    -->
<!-- this work for additional information regarding copyright ownership.      -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0  -->
<!-- (the "License"); you may not use this file except in compliance with     -->
<!-- the License.  You may obtain a copy of the License at                    -->
<!--                                                                          -->
<!--     http://www.apache.org/licenses/LICENSE-2.0                           -->
<!--                                                                          -->
<!-- Unless required by applicable law or agreed to in writing, software      -->
<!-- distributed under the License is distributed on an "AS IS" BASIS,        -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and      -->
<!-- limitations under the License.                                           -->

<configuration>

<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files  -->
<!-- that are implied by Hadoop setup variables.                                                -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive    -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource).                                                                                 -->

<!-- Hive Execution Parameters -->


<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-10-253-1-19.ec2.internal</value>
  <description>http://wiki.apache.org/hadoop/Hive/HBaseIntegration</description>
</property>

<property>
  <name>hive.execution.engine</name>
  <value>tez</value>
</property>

  <property>
    <name>fs.defaultFS</name>
    <value>alluxio://localhost:19998</value>
  </property>


  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://ip-10-253-1-19.ec2.internal:9083</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://ip-10-253-1-19.ec2.internal:3306/hive?createDatabaseIfNotExist=true</value>
    <description>username to use against metastore database</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>org.mariadb.jdbc.Driver</value>
    <description>username to use against metastore database</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>username to use against metastore database</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>[REMOVED]</value>
    <description>password to use against metastore database</description>
  </property>

  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>true</value>
  </property>

  <property>
    <name>mapred.reduce.tasks</name>
    <value>-1</value>
  </property>

  <property>
    <name>mapred.max.split.size</name>
    <value>256000000</value>
  </property>

  <property>
    <name>hive.metastore.connect.retries</name>
    <value>15</value>
  </property>

  <property>
    <name>hive.optimize.sort.dynamic.partition</name>
    <value>true</value>
  </property>

</configuration>

On Wednesday, January 25, 2017 at 5:36:41 AM UTC-5, 黄志 wrote:

Can you provide all the properties in hive-site.xml?Because you init hive with dbType = derby but it complaints that it's not a MYSQL syntax.And the Tez error seems that your HDFS URI is incomplete(I think it should be hdfs://host:port, not hdfs:///).

在 2017年1月21日星期六 UTC+8上午4:08:10,Dan写道:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

John C
In reply to this post by Dan
Was able to get past this by editing /etc/tez/tez-site.xml <name>tez.lib.uris</name> value (it had something like hdfs:///apps...).

Apparently Tez is sometimes used by hive.


On Friday, January 20, 2017 at 3:08:10 PM UTC-5, Dan wrote:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Yufa Zhou
In reply to this post by Dan
Hi Dan,

Have you resolved the problem?

On Saturday, January 21, 2017 at 4:08:10 AM UTC+8, Dan wrote:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Boe H
In reply to this post by Dan
Hi Dan - Would be great if to hear an update.

Also for anyone who can answer this - I noticed there are 2 different versions of the jar file referenced depending on if you are reading docs on alluxio.com or alluxio.org:

1) alluxio-community-1.3.0-hadoop-client.jar
2) alluxio-core-client-1.3.0-jar-with-dependencies.jar

What exactly is the difference between these two libraries? I haven't had success with #2 on Hive/EMR but may give #1 a try.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Yufa Zhou
Dear Boe,

You have set Connection URL to the mysql database in hive-site.xml, which means that you want to store hive metadata in mysql. However, when you initializing hive scheme, you specified -dbType=derby, which is inconsistent with hive-site.xml. You can try to specify -dbType=mysql.

On Wednesday, February 15, 2017 at 12:08:30 AM UTC+8, Boe H wrote:
Hi Dan - Would be great if to hear an update.

Also for anyone who can answer this - I noticed there are 2 different versions of the jar file referenced depending on if you are reading docs on <a href="http://alluxio.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBX05zcSsMVKyYu5VW-WXWhLx2Sg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBX05zcSsMVKyYu5VW-WXWhLx2Sg&#39;;return true;">alluxio.com or <a href="http://alluxio.org" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHWaypJTlImXRAfg_gJTYxVUj9MzQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.org\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHWaypJTlImXRAfg_gJTYxVUj9MzQ&#39;;return true;">alluxio.org:

1) alluxio-community-1.3.0-hadoop-client.jar
2) alluxio-core-client-1.3.0-jar-with-dependencies.jar

What exactly is the difference between these two libraries? I haven't had success with #2 on Hive/EMR but may give #1 a try.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Yufa Zhou
In reply to this post by Dan
Hi Dan,

Have you resolved the problem?

Best

On Saturday, January 21, 2017 at 4:08:10 AM UTC+8, Dan wrote:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio with Hive on EMR

Kumar Gadamsetty
In reply to this post by John C
Hi,

I'm facing the same problem. what should the value for tez.lib.uris parameter to be replaced to?

On Thursday, January 26, 2017 at 5:56:13 PM UTC-5, John C wrote:
Was able to get past this by editing /etc/tez/tez-site.xml <name>tez.lib.uris</name> value (it had something like hdfs:///apps...).

Apparently Tez is sometimes used by hive.


On Friday, January 20, 2017 at 3:08:10 PM UTC-5, Dan wrote:
All,

Some quick notes about running Alluxio on Hive:

My Environment is as follows:
AWS EMR Cluster
EMR Version: emr-5.1.0
EMR Applications: Hive, Ganglia, Spark
Master Instance: r3.xlarge
Core Instances: r3.xlarge (4x)
Alluxio Version: 1.5.0-SNAPSHOT
Hadoop Version: 2.7.3-amzn-0
OS: Amazon Linux (based off of RHEL)
Java Version: openjdk version "1.8.0_111"

- I followed this guide: <a onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.com%2Fdocs%2Fcommunity%2F1.3%2Fen%2FRunning-Hive-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH9VxUQDm6-o_W7G_N3Xupl5bVhwQ&#39;;return true;" href="https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html" target="_blank" rel="nofollow">https://www.alluxio.com/docs/community/1.3/en/Running-Hive-on-Alluxio.html
- On EMR, protobuf-2.5.0 is used which caused conflicts with the below JAR using 2.6.1. In the main POM, protobuf.version is a variable and can be set. I had to change the subsequent POMs to allow the version to be set to 2.5.0 (which is inherited from the main pom). So I built the project using the following:
mvn clean install -Pyarn -DskipTests=true -Dhadoop.version=2.7.3 -Djava.version=1.8
- This path, /<PATH_TO_ALLUXIO>/client/hadoop/alluxio-community-1.3.0-hadoop-client.jar doesn't exist and I couldn't find this Jar anywhere. Instead, I used HADOOP_CLASSPATH="/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}
- On EMR, HIVE_HOME is /usr/lib/hive
- Copy /<PATH_TO_ALLUXIO>/core/client/target/*.jar to $HIVE_HOME
- Hive seems to expect HIVE_INSTALL_DIR to be set; set this to $HIVE_HOME
- Add the following Classifications to your EMR setup (represented in CF):
         {
            "Classification": "hadoop-env",
            "Configurations": [
              {
                "Classification": "export",
                "ConfigurationProperties": {
                  "HADOOP_CLASSPATH" : "\"/<PATH_TO_ALLUXIO>/core/client/target/*:${HADOOP_CLASSPATH}\"",
                  "HIVE_INSTALL_DIR" : "/usr/lib/hive"
                },
                "Configurations": [ ]
              }
            ]
          },
          {
            "Classification" : "hive-site",
            "ConfigurationProperties" : {
              "fs.defaultFS" : "alluxio://localhost:19998"
             },
            "Configurations" : [ ]
          },
          {
            "Classification" : "core-site",
            "ConfigurationProperties" : {
              "fs.alluxio.impl" : "alluxio.hadoop.FileSystem",
              "fs.AbstractFileSystem.alluxio.impl" : "alluxio.hadoop.AlluxioFileSystem"
             },
            "Configurations" : [ ]
          }
- Note: alluxio://localhost:19998 should probably be the actual host name of the instance. I haven't figured out a "global" property that I could derive this from yet.
- ${HIVE_HOME}/bin/schematool -initSchema -dbType derby
    * I haven't been able to get this step to work on EMR, I get the following error:

Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE ' at line 1

Query is:

CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' (state=42000,code=1064)

org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

Underlying cause: java.io.IOException : Schema script failed, errorcode 2

- CREATE TABLE example does work
- I get the following error when I launch the hive CLI. I think there's a property missing:

ERROR tez.TezSessionState: Failed to start Tez session

java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz

at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)

at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:217)

at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)

at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)

at org.apache.tez.client.TezClient.start(TezClient.java:447)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:96)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:327)

at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:323)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.lang.Thread.run(Thread.java:745)

- I get the following warning when I exit the Hive cli:

WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.NullPointerException

java.lang.NullPointerException

at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:228)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:178)

at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:172)

at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:321)

at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)

at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1409)

at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2070)

at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:134)

at alluxio.hadoop.FileSystem.close(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2760)

at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2777)

at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

-

Questions:
Should the protobuf-java module have a parameterized version (<version>${protobuf.version}</version>) in all POMs (ie core/client/pom.xml)?
Any ideas on what's causing the above errors?

Thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.