Run Alluxio with PIG

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Run Alluxio with PIG

Mania
Hello

I am running PIG + Hadoop on top of Alluxio and I am using HDFS as underfs storage. I use

alluxio://<alluxio_host>:<alluxio_port>/pigmix/ as the address for input data.

The problem I have: Alluxio doesnot write data back to the UnderFS storage and I got LOST error for the evicted data.

To deploy Alluxio, I followed the use setup explained in the Alluxio documentation and I use synchronous mode for eviction.

Alluxio Version: 1.8.0
Hadoop Version: 2.8
PIG version: 0.17

Any help is appreciated to resolve this issue.



Regards
Mania

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run Alluxio with PIG

Bin Fan
hi Mania,

for 1.8.0, default write type is MUST_CACHE which is writing to Alluxio only.

To write data to both Alluxio and under store:
if you want to set it for a job, you can either set the write type to CACHE_THROUGH  
like this: alluxio.user.file.writetype.default=CACHE_THROUGH
though I am not sure how to set conf in PIG,

Alternatively, if you want to set the write type globally to all Alluxio accessing jobs 
(including your PIG jobs and potentially other Spark/MR jobs)
you can make it cluster-default.

- Bin


On Sun, Oct 21, 2018 at 12:18 PM Mania <[hidden email]> wrote:
Hello

I am running PIG + Hadoop on top of Alluxio and I am using HDFS as underfs storage. I use

alluxio://<alluxio_host>:<alluxio_port>/pigmix/ as the address for input data.

The problem I have: Alluxio doesnot write data back to the UnderFS storage and I got LOST error for the evicted data.

To deploy Alluxio, I followed the use setup explained in the Alluxio documentation and I use synchronous mode for eviction.

Alluxio Version: 1.8.0
Hadoop Version: 2.8
PIG version: 0.17

Any help is appreciated to resolve this issue.



Regards
Mania

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.