Hive statements failing with alluxio files having space in their name(Alluxio 1.7.1)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Hive statements failing with alluxio files having space in their name(Alluxio 1.7.1)

Omkar Naidu
Hi Team,

As per release notes of alluxio 1.7.1 Alluxio Support white space in file names.

I am able to create,list,copy and delete files with spaces but hive statments for failing on alluxio with space in file names.

Below is the error I am getting when trying insert rows into hive table with alluxio as location.

Same query works if no space in file name.Is there any settings I need to take care to use hive with alluxio having space in file names/this is a known issue?

----table creation-----
hive> CREATE TABLE alluxio.test (
    > NAME STRING,
    > SEX CHAR(1),
    > AGE INT,
    > Height_in INT,
    > Weight_lbs INT)
    > ROW FORMAT DELIMITED
    > FIELDS TERMINATED BY ','
    > LOCATION "bdfs://localhost:19998/test table";
OK
Time taken: 2.535 seconds


-------------------------------------------

-----insert statement and error-----------
hive> insert into table alluxio.test values('alluxio1','f',2,68,129),('alluxio12','f',2,69,125),('alluxio13','f',2,70,127);
Query ID = oracle_20180928064212_ae5895c1-091c-41f2-b4ad-9996f12ff700
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1538114221407_0004)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED      1          1        0        0       2       0
----------------------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 27.66 s
----------------------------------------------------------------------------------------------
Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Unable to move: bdfs://localhost:19998/test table/.hive-staging_hive_2018-09-28_06-42-12_872_8356234333893709918-1/_tmp.-ext-10000 to: bdfs://localhost:19998/test table/.hive-staging_hive_2018-09-28_06-42-12_872_8356234333893709918-1/-ext-10000)'
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
-------------------------------------------------------------------------------------------------------

Regards
Omkar

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive statements failing with alluxio files having space in their name(Alluxio 1.7.1)

Bin Fan
hi Omkar,

Just curious, is bdfs equivalent to alluxio?

I tested alluxuio 1.7.1, I was able to create and rename files with space in their names.
Can you maybe provide more detailed job failure logs? or checkout the Alluxio master log?

[14:57:57]binfan@:v1.7 ((v1.7.1))$ bin/alluxio fs mkdir "/name with space"

Successfully created directory /name with space

[14:58:07]binfan@:v1.7 ((v1.7.1))$ bin/alluxio fs ls /

drwxr-xr-x binfan         staff                        0   NOT_PERSISTED 09-28-2018 14:58:07:366  DIR /name with space

[14:58:13]binfan@:v1.7 ((v1.7.1))$ bin/alluxio fs mv "/name with space" "/new name with space"

Renamed /name with space to /new name with space

[14:58:36]binfan@:v1.7 ((v1.7.1))$ bin/alluxio fs ls /

drwxr-xr-x binfan         staff                        0   NOT_PERSISTED 09-28-2018 14:58:07:366  DIR /new name with space


On Thu, Sep 27, 2018 at 11:49 PM Omkar Naidu <[hidden email]> wrote:
Hi Team,

As per release notes of alluxio 1.7.1 Alluxio Support white space in file names.

I am able to create,list,copy and delete files with spaces but hive statments for failing on alluxio with space in file names.

Below is the error I am getting when trying insert rows into hive table with alluxio as location.

Same query works if no space in file name.Is there any settings I need to take care to use hive with alluxio having space in file names/this is a known issue?

----table creation-----
hive> CREATE TABLE alluxio.test (
    > NAME STRING,
    > SEX CHAR(1),
    > AGE INT,
    > Height_in INT,
    > Weight_lbs INT)
    > ROW FORMAT DELIMITED
    > FIELDS TERMINATED BY ','
    > LOCATION "bdfs://localhost:19998/test table";
OK
Time taken: 2.535 seconds


-------------------------------------------

-----insert statement and error-----------
hive> insert into table alluxio.test values('alluxio1','f',2,68,129),('alluxio12','f',2,69,125),('alluxio13','f',2,70,127);
Query ID = oracle_20180928064212_ae5895c1-091c-41f2-b4ad-9996f12ff700
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1538114221407_0004)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED      1          1        0        0       2       0
----------------------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 27.66 s
----------------------------------------------------------------------------------------------
Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Unable to move: bdfs://localhost:19998/test table/.hive-staging_hive_2018-09-28_06-42-12_872_8356234333893709918-1/_tmp.-ext-10000 to: bdfs://localhost:19998/test table/.hive-staging_hive_2018-09-28_06-42-12_872_8356234333893709918-1/-ext-10000)'
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
-------------------------------------------------------------------------------------------------------

Regards
Omkar

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.