Hive error -- No FileSystem for scheme: alluxio

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Lu Qiu
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Lu Qiu
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf fs.default.name=alluxio://localhost:19998 when invoking Hive, fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


<a imageanchor="1" href="about:invalid#zClosurez" style="margin-left: 1em; margin-right: 1em;">core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf <a href="http://fs.default.name/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;">fs.default.name=alluxio://localhost:19998 when invoking Hive, <a href="http://fs.default.name" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in <a href="http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;">http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2082qDUUBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kumar.gad...@...> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2082qDUUBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">l...@...> wrote:
Hi Kumar,

Could you follow the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html
and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(<a href="https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2082qDUUBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kumar.gad...@...> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf <a href="http://fs.default.name" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2082qDUUBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
I kept wrong syntax for table creation in my previous post:

18/10/18 15:50:42 INFO ql.Driver: Executing command(queryId=ec2-user_20181018155042_7b322baf-c78d-42f0-9b1b-d604cded291c): CREATE  TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.dim_rdm_batch on alluxio://localhost:19998/user/hive/warehouse/t1



On Thursday, October 18, 2018 at 11:55:42 AM UTC-4, Kumar Gadamsetty wrote:
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf <a href="http://fs.default.name/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;">fs.default.name=alluxio://localhost:19998 when invoking Hive, <a href="http://fs.default.name" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in <a href="http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;">http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

Could you follow the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html
and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(<a href="https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf <a href="http://fs.default.name" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Lu Qiu
Hi Kumar

Thanks for your previous efforts to confirm that Hive can recognize Alluxio but Hive metastore cannot.
Hive metastore also needs to find alluxio client jar. Alluxio client jar needs to be put on the same machine as hive metastore, and need to be put in the classpath of hive metastore.
After those changes, hive metastore needs to be restarted to pick up all the changes.

Could you provide us more information about your hive metastore? 

Thanks,
Lu

On Thu, Oct 18, 2018 at 8:58 AM Kumar Gadamsetty <[hidden email]> wrote:
I kept wrong syntax for table creation in my previous post:

18/10/18 15:50:42 INFO ql.Driver: Executing command(queryId=ec2-user_20181018155042_7b322baf-c78d-42f0-9b1b-d604cded291c): CREATE  TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.dim_rdm_batch on alluxio://localhost:19998/user/hive/warehouse/t1



On Thursday, October 18, 2018 at 11:55:42 AM UTC-4, Kumar Gadamsetty wrote:
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf fs.default.name=alluxio://localhost:19998 when invoking Hive, fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
T

On Thu, Oct 18, 2018 at 2:30 PM Lu <[hidden email]> wrote:
Hi Kumar

Thanks for your previous efforts to confirm that Hive can recognize Alluxio but Hive metastore cannot.
Hive metastore also needs to find alluxio client jar. Alluxio client jar needs to be put on the same machine as hive metastore, and need to be put in the classpath of hive metastore.
After those changes, hive metastore needs to be restarted to pick up all the changes.

Could you provide us more information about your hive metastore? 

Thanks,
Lu

On Thu, Oct 18, 2018 at 8:58 AM Kumar Gadamsetty <[hidden email]> wrote:
I kept wrong syntax for table creation in my previous post:

18/10/18 15:50:42 INFO ql.Driver: Executing command(queryId=ec2-user_20181018155042_7b322baf-c78d-42f0-9b1b-d604cded291c): CREATE  TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.dim_rdm_batch on alluxio://localhost:19998/user/hive/warehouse/t1



On Thursday, October 18, 2018 at 11:55:42 AM UTC-4, Kumar Gadamsetty wrote:
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf fs.default.name=alluxio://localhost:19998 when invoking Hive, fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Kumar Gadamsetty
In reply to this post by Lu Qiu
Yes I missed copying jar file to hive lib directory and restart hive-server2 and hive catalog. Now it works.

Thanks LU and appreciate for the help.



On Thursday, October 18, 2018 at 2:30:51 PM UTC-4, Lu wrote:
Hi Kumar

Thanks for your previous efforts to confirm that Hive can recognize Alluxio but Hive metastore cannot.
Hive metastore also needs to find alluxio client jar. Alluxio client jar needs to be put on the same machine as hive metastore, and need to be put in the classpath of hive metastore.
After those changes, hive metastore needs to be restarted to pick up all the changes.

Could you provide us more information about your hive metastore? 

Thanks,
Lu

On Thu, Oct 18, 2018 at 8:58 AM Kumar Gadamsetty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ELuCOHdYBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kumar.gad...@...> wrote:
I kept wrong syntax for table creation in my previous post:

18/10/18 15:50:42 INFO ql.Driver: Executing command(queryId=ec2-user_20181018155042_7b322baf-c78d-42f0-9b1b-d604cded291c): CREATE  TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.dim_rdm_batch on alluxio://localhost:19998/user/hive/warehouse/t1



On Thursday, October 18, 2018 at 11:55:42 AM UTC-4, Kumar Gadamsetty wrote:
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf <a href="http://fs.default.name/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFr6plTm77K2LbsAYcz25NhNxSaw&#39;;return true;">fs.default.name=alluxio://localhost:19998 when invoking Hive, <a href="http://fs.default.name" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in <a href="http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHB17cFfVeNOJNVnxukhSPphq--MA&#39;;return true;">http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

Could you follow the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hive-with-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFIhtX8dQ-6xxOmt0UR6Hdu0gChjA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html
and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(<a href="https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Hadoop-MapReduce-on-Alluxio.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE70dl5d5-kmL79UJexOnM9jKifuA&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf <a href="http://fs.default.name" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffs.default.name\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFLGTK29rZEUMeVGojX5gzRP9xJtg&#39;;return true;">fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ELuCOHdYBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive error -- No FileSystem for scheme: alluxio

Lu Qiu
Glad that it works!

Thanks for the confirmation! :)


On Thu, Oct 18, 2018 at 1:09 PM Kumar Gadamsetty <[hidden email]> wrote:
Yes I missed copying jar file to hive lib directory and restart hive-server2 and hive catalog. Now it works.

Thanks LU and appreciate for the help.



On Thursday, October 18, 2018 at 2:30:51 PM UTC-4, Lu wrote:
Hi Kumar

Thanks for your previous efforts to confirm that Hive can recognize Alluxio but Hive metastore cannot.
Hive metastore also needs to find alluxio client jar. Alluxio client jar needs to be put on the same machine as hive metastore, and need to be put in the classpath of hive metastore.
After those changes, hive metastore needs to be restarted to pick up all the changes.

Could you provide us more information about your hive metastore? 

Thanks,
Lu

On Thu, Oct 18, 2018 at 8:58 AM Kumar Gadamsetty <[hidden email]> wrote:
I kept wrong syntax for table creation in my previous post:

18/10/18 15:50:42 INFO ql.Driver: Executing command(queryId=ec2-user_20181018155042_7b322baf-c78d-42f0-9b1b-d604cded291c): CREATE  TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.dim_rdm_batch on alluxio://localhost:19998/user/hive/warehouse/t1



On Thursday, October 18, 2018 at 11:55:42 AM UTC-4, Kumar Gadamsetty wrote:
Hi Lu,

I configured the parameters and integration check also succeeded.


***** The integration checker ran at 18/10/18 15:32:56. *****

Nodes of IP addresses: 10.xx.xxx.xx 10.xx.xxx.xx can recognize Alluxio filesystem.

***** Integration test passed. *****


core-site.JPG


Also added the below property in /etc/hive/conf/hive-site.xml
<property>
   <name>fs.defaultFS</name>
   <value>alluxio://localhost:19998</value>
</property>


Now when I try to create table on hive, I'm getting this error:

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';

18/10/18 15:50:42 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
18/10/18 15:50:42 INFO exec.DDLTask: creating table default.t1 on alluxio://localhost:19998/user/hive/warehouse/t1
18/10/18 15:50:42 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 


On Wednesday, October 17, 2018 at 5:40:02 PM UTC-4, Lu wrote:
Hi Kumar,

If you want to set alluxio as the default filesystem of Hive, 
you could add the following property to `hive-site.xml`:

<property>
   <name>fs.defaultFS</name>
   <value>alluxio://master_hostname:port</value>
</property>
or you can use hive --hiveconf fs.defaultFS=alluxio://localhost:19998
I notice that you used --hiveconf fs.default.name=alluxio://localhost:19998 when invoking Hive, fs.default.name is deprecated.

In addition, if you want to create external table in Hive, you do not need to set alluxio as the default filesystem of Hive.
In most cases, HDFS is still the default filesystem of Hive and Hive needs to invoke MapReduce to execute the query. Could you try to see if the MapReduce example in http://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html works? If not, please make sure 

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
is set in the core-site.xml file of your Hadoop installation. 

Could you share us more information? How many nodes are your Hive server running? Did you run your metastore server on the same node as Hive Server. How many hive-site.xml do you have?

Thanks,
Lu


On Wed, Oct 17, 2018 at 1:31 PM Kumar Gadamsetty <[hidden email]> wrote:
I followed the same links and configured all the files.

On Wed, Oct 17, 2018 at 4:01 PM Lu <[hidden email]> wrote:
Hi Kumar,

and see if the create a new external table works?

Please note that to run Hive on Hadoop MapReduce, you also need to follow the instructions in running MapReduce on Alluxio(https://www.alluxio.org/docs/1.8/en/Running-Hadoop-MapReduce-on-Alluxio.html) to make sure Hadoop MapReduce can run with Alluxio.

Thanks,
Lu




On Wed, Oct 17, 2018 at 11:39 AM Kumar Gadamsetty <[hidden email]> wrote:
Version: Alluxio 1.8.0
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Hive 2.3.2-amzn-2
EMR 5.14.0

I was able to load data into management table from alluxio successfully but unable to create external table(as below) in hive using alluxio location file.

CREATE EXTERNAL TABLE t1(
  batch_base_id bigint,
  engine_run_timestamp_utc timestamp)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS orc
LOCATION 'alluxio://localhost:19998/mnt/s3/t1';
   
But the hive is unable to recognize alluxio file system and failing with the below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
18/10/17 18:19:47 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)

Tried specifying all the below parameter when invoking hive but nothing works:
hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.default.name=alluxio://localhost:19998

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.