Writing data throught proxy generate errors (Rest Endpoint filename allready exists)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing data throught proxy generate errors (Rest Endpoint filename allready exists)

Autaa François
Alluxio 1.7.1-hadoop-2.8
Mesos 1.4.1
Ceph Luminous for DFS and Object Storage with S3 interface 
Centos 7.3.1611
JDK 1.8.0_131
Spark 2.2.1
Boto3/Botocore


Hi

I still try to stress the proxy and see how he can handle large amount of data. Now I try to load ~100Gb of data using threaded pyhton ( 100 thread with 1 thread per file )

But I face some issue. The default behaviour I would like is to have both alluxio & s3 data in sync.
So I logically set alluxio.proxy.s3.writetype=CACHE_THROUGH

I ran my python code and proxy logs keep saying

WARN S3RestUtils - Unexpected error invoking rest endpoint : myfilename already exists.


Despite this error the final checksum in my S3 is okay. I've tried differents things into property file to remove this warning but unable to remove it.

Then I've tried to set alluxio.proxy.s3.writetype=ASYNC_THROUGH and then my write operation works fine ( good size under alluxio ) but I'm absolutly not in sync in S3 even 15 minutes later.

No idea about what I'm doing wrong here ..

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Writing data throught proxy generate errors (Rest Endpoint filename allready exists)

Gene Pang
Hi,

How are the threads writing the data to the proxy? Also, do you see any messages in the Alluxio master when using ASYNC_THROUGH?

Thanks,
Gene

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.