Hive with alluxio, "map existing tables stored in hdfs to alluxio" can't cached partitioned table.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Hive with alluxio, "map existing tables stored in hdfs to alluxio" can't cached partitioned table.

Xianglin Chen
Hi,
    After configuring hive with alluxio, I wanted to map existing tables stored in hdfs to alluxio. The commands "alter table xxx set location 'alluxio://xxx'" are succeeded. The information of "desc formatted xxx" is right. 

    I accessed files for the usual tables using the command "select * for table_name", files cached in alluxio successfully.

    But for partition tables in hive, after the command "select * for table_name", the files didn't cached in alluxio.

    There isn't any errors in logs, the document "http://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html#map-existing-tables-stored-in-hdfs-to-alluxio" said that "the entire process is transparent to Hive and users". So I don't know how to solve the problem and ask for help.
    My configuration:
    alluxio 1.8.0 build with hadoop 3.0.3.
    hive 3.0.0.
    hadoop 3.0.3.
    tables are generated by "https://github.com/hortonworks/hive-testbench"
thanks.
Xianglin Chen

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive with alluxio, "map existing tables stored in hdfs to alluxio" can't cached partitioned table.

Thai Bui
For partitioned table in hive with alluxio you’ll have to alter the partition’s locations with the alter DDL. For example, 

alter table your_table partition (partition_col=value) set location ‘alluxio://..’

On Wed, Aug 15, 2018 at 9:38 AM Xianglin Chen <[hidden email]> wrote:
Hi,
    After configuring hive with alluxio, I wanted to map existing tables stored in hdfs to alluxio. The commands "alter table xxx set location 'alluxio://xxx'" are succeeded. The information of "desc formatted xxx" is right. 

    I accessed files for the usual tables using the command "select * for table_name", files cached in alluxio successfully.

    But for partition tables in hive, after the command "select * for table_name", the files didn't cached in alluxio.

    There isn't any errors in logs, the document "http://www.alluxio.org/docs/1.8/en/Running-Hive-with-Alluxio.html#map-existing-tables-stored-in-hdfs-to-alluxio" said that "the entire process is transparent to Hive and users". So I don't know how to solve the problem and ask for help.
    My configuration:
    alluxio 1.8.0 build with hadoop 3.0.3.
    hive 3.0.0.
    hadoop 3.0.3.
    tables are generated by "https://github.com/hortonworks/hive-testbench"
thanks.
Xianglin Chen

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--
Thai

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive with alluxio, "map existing tables stored in hdfs to alluxio" can't cached partitioned table.

Gene Pang
Hi Xianglin,

Were you able to resolve the issue?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Hive with alluxio, "map existing tables stored in hdfs to alluxio" can't cached partitioned table.

Xianglin Chen
Yes, thanks!
I'm so sorry that I forgot to reply.

在 2018年8月28日星期二 UTC+8上午6:38:51,Gene Pang写道:
Hi Xianglin,

Were you able to resolve the issue?

Thanks,
Gene

<a href="http://bit.ly/2EmpC7u" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2EmpC7u\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG_0jMFTBLczk38hwD4XXX91KWwIQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2EmpC7u\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG_0jMFTBLczk38hwD4XXX91KWwIQ&#39;;return true;">alluxio.com | <a href="http://bit.ly/2G7XIIO" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2G7XIIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOybm0pd7v3PlGRUU-joWOqGBDQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2G7XIIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOybm0pd7v3PlGRUU-joWOqGBDQw&#39;;return true;">alluxio.org <a href="http://bit.ly/2JD5Cwk" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2JD5Cwk\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9U1Eocd-eCYZPelgbTpJ1dZJucQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2F2JD5Cwk\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9U1Eocd-eCYZPelgbTpJ1dZJucQ&#39;;return true;">powered by Alluxio

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.