Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

Xiaohui Liu
Hi, Gurus

I am a new users of alluxio. I am using S3 as the ufs in alluxio. I had two issues that need your help:

1. S3 server-side-encryption
    Server-side encryption is enabled in most of our buckets. jets3t has a setting to enable this (otherwise, one cannot write the buckets successfully), s3service.server-side-encryption=AES256. But I am not sure if this property is applicable in alluxio nor how to enable it in alluxio. 
    I tried to create a jets3t.properties file under 'conf'. But it does not seem to work.

2. Using IAM instance profile instead of fixed credentials
    I tried 1.1.0-snapshot with this patch merged (https://github.com/Alluxio/alluxio/pull/3043). It does not report credentials' missing errors, but it still complains about objects do not exist, 
e.g.
    ThriftIOException(message:Ufs path <S3_PATH> does not exist)
    
Has anyone encounter similar issues? It would be greatly appreciated if you'd like to share you experience. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

binfan
Administrator
Hi, Xiaohui

For jets3t property "s3service.server-side-encryption", it is not currently
picked by S3 under storage integration yet. it seems to me a good thing to
have.

Alluxio devs, what do you think to respect all "s3service.*"?

- Bin

 



--
View this message in context: http://alluxio-users.85194.x6.nabble.com/Problems-in-using-S3-as-the-under-filesystem-server-side-encryption-IAM-instance-profile-tp3p4.html
Sent from the Alluxio Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

Bin Fan
In reply to this post by Xiaohui Liu
Hi, Xiaohui

For jets3t property "s3service.server-side-encryption", it is not currently picked by S3 under storage integration yet. it seems to me a good thing to have.

Alluxio devs, what do you think to respect all "s3service.*"?

- Bin
Reply | Threaded
Open this post in threaded view
|

Re: Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

Chaomin Yu
In reply to this post by Xiaohui Liu
Hi Xiaohui,

Thanks for your interests in Alluxio.

1. Alluxio does not support your customized jets3t.properties directly. Current supported S3 configurations are described here
It makes sense to support server-side encryption, I have a PR out to support this feature: https://github.com/Alluxio/alluxio/pull/3314
Can you try and see if it work for you?

I tested it with a server-side encrypted bucket:
Inline image 1



2. Can you please provide some more details (i.e. attach logs under {ALLUXIO_HOME}/logs/ ) about the object non exist error?  thanks!

Hope this helps,
Chaomin

On Wed, May 18, 2016 at 1:04 AM, Xiaohui Liu <[hidden email]> wrote:
Hi, Gurus

I am a new users of alluxio. I am using S3 as the ufs in alluxio. I had two issues that need your help:

1. S3 server-side-encryption
    Server-side encryption is enabled in most of our buckets. jets3t has a setting to enable this (otherwise, one cannot write the buckets successfully), s3service.server-side-encryption=AES256. But I am not sure if this property is applicable in alluxio nor how to enable it in alluxio. 
    I tried to create a jets3t.properties file under 'conf'. But it does not seem to work.

2. Using IAM instance profile instead of fixed credentials
    I tried 1.1.0-snapshot with this patch merged (https://github.com/Alluxio/alluxio/pull/3043). It does not report credentials' missing errors, but it still complains about objects do not exist, 
e.g.
    ThriftIOException(message:Ufs path <S3_PATH> does not exist)
    
Has anyone encounter similar issues? It would be greatly appreciated if you'd like to share you experience. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems in using S3 as the under filesystem [server-side-encryption & IAM instance profile]

Xiaohui Liu
In reply to this post by Bin Fan
Hi, I did not get time to test the merged code until today. server-side-encryption works very well. Now I am able to mount s3 buckets with sse enabled to Alluxio.

But the latest master branch still does not work with my instances profile. I have granted full bucket permission to the underlining IAM role. While attempting to mount a directory from that bucket, master.log shows the following info. The target <BUCKET>/libs/gqt/ does exist and has two files inside. But for some reason, Alluxio cannot find the keys.

Any suggestions?


2016-05-25 05:04:24,647 DEBUG service.Jets3tProperties (Jets3tProperties.java:getBoolProperty) - s3service.disable-dns-buckets=true
2016-05-25 05:04:24,647 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,647 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint-virtual-path=
2016-05-25 05:04:24,648 DEBUG service.Jets3tProperties (Jets3tProperties.java:getIntProperty) - s3service.s3-endpoint-https-port=443
2016-05-25 05:04:24,648 DEBUG httpclient.RestStorageService (RestStorageService.java:setupConnection) - S3 URL: https://s3.amazonaws.com:443/<BUCKET>/
2016-05-25 05:04:24,648 DEBUG httpclient.RestStorageService (RestStorageService.java:addRequestParametersToUrlPath) - Added request parameter: max-keys=1000
2016-05-25 05:04:24,648 DEBUG httpclient.RestStorageService (RestStorageService.java:addRequestParametersToUrlPath) - Added request parameter: prefix=libs/gqt/
2016-05-25 05:04:24,648 DEBUG httpclient.RestStorageService (RestStorageService.java:addRequestParametersToUrlPath) - Added request parameter without value: delimiter
2016-05-25 05:04:24,649 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Performing GET request for 'https://s3.amazonaws.com:443/<BUCKET>/?max-keys=1000&prefix=libs%2Fgqt%2F&delimiter', expecting response codes: [200]
2016-05-25 05:04:24,649 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Headers: [Date: Wed, 25 May 2016 05:04:24 GMT]
2016-05-25 05:04:24,649 DEBUG service.Jets3tProperties (Jets3tProperties.java:getIntProperty) - httpclient.retry-max=5
2016-05-25 05:04:24,649 DEBUG httpclient.RestStorageService (RestStorageService.java:authorizeHttpRequest) - Adding authorization for Access Key ''.
2016-05-25 05:04:24,650 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,650 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - storage-service.request-signature-version=AWS2
2016-05-25 05:04:24,650 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,650 DEBUG httpclient.RestStorageService (RestStorageService.java:authorizeHttpRequest) - Canonical string ('|' is a newline): GET|||Wed, 25 May 2016 05:04:24 GMT|/<BUCKET>/
2016-05-25 05:04:24,651 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Rethrowing as a ServiceException error in performRequest: java.lang.IllegalArgumentException: Empty key
2016-05-25 05:04:24,651 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Releasing HttpClient connection after error: Empty key
2016-05-25 05:04:24,651 DEBUG httpclient.RestStorageService (RestStorageService.java:getObjectImpl) - Retrieving Head information for bucket <BUCKET> and object libs/gqt
2016-05-25 05:04:24,651 DEBUG service.Jets3tProperties (Jets3tProperties.java:getBoolProperty) - s3service.disable-dns-buckets=true
2016-05-25 05:04:24,651 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,652 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint-virtual-path=
2016-05-25 05:04:24,652 DEBUG service.Jets3tProperties (Jets3tProperties.java:getIntProperty) - s3service.s3-endpoint-https-port=443
2016-05-25 05:04:24,652 DEBUG httpclient.RestStorageService (RestStorageService.java:setupConnection) - S3 URL: https://s3.amazonaws.com:443/<BUCKET>/libs/gqt
2016-05-25 05:04:24,652 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Performing HEAD request for 'https://s3.amazonaws.com:443/<BUCKET>/libs/gqt', expecting response codes: [200]
2016-05-25 05:04:24,652 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Headers: [Date: Wed, 25 May 2016 05:04:24 GMT]
2016-05-25 05:04:24,653 DEBUG service.Jets3tProperties (Jets3tProperties.java:getIntProperty) - httpclient.retry-max=5
2016-05-25 05:04:24,653 DEBUG httpclient.RestStorageService (RestStorageService.java:authorizeHttpRequest) - Adding authorization for Access Key ''.
2016-05-25 05:04:24,653 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,653 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - storage-service.request-signature-version=AWS2
2016-05-25 05:04:24,653 DEBUG service.Jets3tProperties (Jets3tProperties.java:getStringProperty) - s3service.s3-endpoint=s3.amazonaws.com
2016-05-25 05:04:24,654 DEBUG httpclient.RestStorageService (RestStorageService.java:authorizeHttpRequest) - Canonical string ('|' is a newline): HEAD|||Wed, 25 May 2016 05:04:24 GMT|/<BUCKET>/libs/gqt
2016-05-25 05:04:24,654 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Rethrowing as a ServiceException error in performRequest: java.lang.IllegalArgumentException: Empty key
2016-05-25 05:04:24,654 DEBUG httpclient.RestStorageService (RestStorageService.java:performRequest) - Releasing HttpClient connection after error: Empty key


On Friday, May 20, 2016 at 12:09:21 AM UTC+8, Bin Fan wrote:
Hi, Xiaohui

For jets3t property "s3service.server-side-encryption", it is not currently
picked by S3 under storage integration yet. it seems to me a good thing to
have.

Alluxio devs, what do you think to respect all "s3service.*"?

- Bin



--
View this message in context: <a href="http://alluxio-users.85194.x6.nabble.com/Problems-in-using-S3-as-the-under-filesystem-server-side-encryption-IAM-instance-profile-tp3p5.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio-users.85194.x6.nabble.com%2FProblems-in-using-S3-as-the-under-filesystem-server-side-encryption-IAM-instance-profile-tp3p5.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVkSTg4_P22nbprlnbsFkAOBrOvg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio-users.85194.x6.nabble.com%2FProblems-in-using-S3-as-the-under-filesystem-server-side-encryption-IAM-instance-profile-tp3p5.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVkSTg4_P22nbprlnbsFkAOBrOvg&#39;;return true;">http://alluxio-users.85194.x6.nabble.com/Problems-in-using-S3-as-the-under-filesystem-server-side-encryption-IAM-instance-profile-tp3p5.html
Sent from the Alluxio Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.