Help with Mounting S3 (Getting 400 Bad Request)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with Mounting S3 (Getting 400 Bad Request)

Clayton McClure

I'm new to Alluxio, still getting initial setup working.  Any help would be greatly appreciated.

Alluxio Version:  Current clone (1.9 snapshot)
Env:  AWS in us-west-2
UFS: S3

I've followed the guide for getting it setup using Vagrant and have a cluster setup in EC2.  I have an IAM profile and a test s3 bucket with data in it.  I've validated I can read data in the bucket from the AlluxioMaster:

[ec2-user@AlluxioMaster ~]$ aws s3 ls s3://redacted-bucket/
                           PRE some-data/
                           PRE some-other-data/
                           PRE tmp/

I also copied up a test file and back down from that bucket.

However, when I try to mount the s3 bucket I get a 400 Bad Request:

ec2-user@AlluxioMaster ~]$ /alluxio/bin/alluxio fs mount --readonly alluxio://localhost:19998/mnt/s3 s3a://redacted-bucket/
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 11E74C12CC0B92A8; S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=), S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=


I've tried a whole bunch of different s3 configuration options, such as disabling dns, using https, specifying the s3 endpoint, etc, but I continually get the same error.  

Does anyone have a suggestion for how else I might trouble shoot this?

Thanks,

Clayton

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with Mounting S3 (Getting 400 Bad Request)

adit
Hi Clayton,

Can you add these properties to conf/log4j.properties and restart the Alluxio master:

log4j.logger.com.amazonaws=WARN
log4j.logger.com.amazonaws.request=DEBUG
log4j.logger.org.apache.http.wire=DEBUG

Please share logs/master.log after making the change and retrying the mount.

Adit

On Sunday, August 19, 2018 at 4:10:45 PM UTC-7, Clayton McClure wrote:

I'm new to Alluxio, still getting initial setup working.  Any help would be greatly appreciated.

Alluxio Version:  Current clone (1.9 snapshot)
Env:  AWS in us-west-2
UFS: S3

I've followed the guide for getting it setup using Vagrant and have a cluster setup in EC2.  I have an IAM profile and a test s3 bucket with data in it.  I've validated I can read data in the bucket from the AlluxioMaster:

[ec2-user@AlluxioMaster ~]$ aws s3 ls s3://redacted-bucket/
                           PRE some-data/
                           PRE some-other-data/
                           PRE tmp/

I also copied up a test file and back down from that bucket.

However, when I try to mount the s3 bucket I get a 400 Bad Request:

ec2-user@AlluxioMaster ~]$ /alluxio/bin/alluxio fs mount --readonly alluxio://localhost:19998/mnt/s3 s3a://redacted-bucket/
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 11E74C12CC0B92A8; S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=), S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=


I've tried a whole bunch of different s3 configuration options, such as disabling dns, using https, specifying the s3 endpoint, etc, but I continually get the same error.  

Does anyone have a suggestion for how else I might trouble shoot this?

Thanks,

Clayton

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with Mounting S3 (Getting 400 Bad Request)

Clayton McClure
Thanks for the response.  I made that change, here's the log file around the s3 request:

2018-08-24 18:26:09,100 INFO  AbstractMaster - MetricsMaster: Starting primary master.
2018-08-24 18:26:09,101 INFO  AbstractMaster - BlockMaster: Starting primary master.
2018-08-24 18:26:09,101 INFO  AbstractMaster - FileSystemMaster: Starting primary master.
2018-08-24 18:26:09,101 INFO  DefaultFileSystemMaster - Starting fs master as primary
2018-08-24 18:26:09,183 INFO  UfsJournalLogWriter - Created current log file: UfsJournalFile{location=/alluxio/journal/BlockMaster/v1/logs/0x0-0x7fffffffffffffff, start=0, end=9223372036854775807, isCheckpoint=false}
2018-08-24 18:26:09,197 INFO  UfsJournalLogWriter - Created current log file: UfsJournalFile{location=/alluxio/journal/FileSystemMaster/v1/logs/0x0-0x7fffffffffffffff, start=0, end=9223372036854775807, isCheckpoint=false}
2018-08-24 18:26:09,204 INFO  AbstractMaster - MetaMaster: Starting primary master.
2018-08-24 18:26:09,211 INFO  AlluxioMasterProcess - All masters started
2018-08-24 18:26:09,212 INFO  MetricsSystem - Starting sinks with config: {}.
2018-08-24 18:26:09,221 INFO  log - Logging initialized @22145ms
2018-08-24 18:26:09,222 DEBUG request - Sending Request: HEAD http://redacted-bucket.s3.us-west-2.amazonaws.com // Headers: (User-Agent: aws-sdk-java/1.11.215 Linux/4.14.62-65.117.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/25.181-b13 java/1.8.0_181, amz-sdk-invocation-id: df16dd27-b301-5d6b-8794-bbe494c33627, Content-Type: application/octet-stream, )
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "HEAD /%2F HTTP/1.1[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "Host: redacted-bucket.s3.us-west-2.amazonaws.com[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=/20180824/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3592182a3ab9701096748790f94c7f9b7876b82c6138082f50955600339b8842[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "X-Amz-Date: 20180824T182609Z[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "User-Agent: aws-sdk-java/1.11.215 Linux/4.14.62-65.117.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/25.181-b13 java/1.8.0_181[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "amz-sdk-invocation-id: df16dd27-b301-5d6b-8794-bbe494c33627[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2018-08-24 18:26:09,300 DEBUG wire - http-outgoing-0 >> "[\r][\n]"
2018-08-24 18:26:09,303 DEBUG wire - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
2018-08-24 18:26:09,303 DEBUG wire - http-outgoing-0 << "x-amz-request-id: 644A904D4436DCD5[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "x-amz-id-2: xvDDOUSl5j5IPcENvlE3inKHEI5aSgQ8vHvkipL2eLKnJChvs68tH4XIN+VdK1ZjEyhB8v0de8Q=[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "Date: Fri, 24 Aug 2018 18:26:08 GMT[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "Connection: close[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "Server: AmazonS3[\r][\n]"
2018-08-24 18:26:09,304 DEBUG wire - http-outgoing-0 << "[\r][\n]"
2018-08-24 18:26:09,310 DEBUG request - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 644A904D4436DCD5; S3 Extended Request ID: xvDDOUSl5j5IPcENvlE3inKHEI5aSgQ8vHvkipL2eLKnJChvs68tH4XIN+VdK1ZjEyhB8v0de8Q=), S3 Extended Request ID: xvDDOUSl5j5IPcENvlE3inKHEI5aSgQ8vHvkipL2eLKnJChvs68tH4XIN+VdK1ZjEyhB8v0de8Q=
2018-08-24 18:26:09,337 INFO  Server - jetty-9.2.z-SNAPSHOT
2018-08-24 18:26:09,360 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@1e800aaa{/metrics/prometheus,null,AVAILABLE}
2018-08-24 18:26:09,361 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@6f03482{/metrics/json,null,AVAILABLE}
2018-08-24 18:26:12,330 INFO  ContextHandler - Started o.e.j.w.WebAppContext@1a1da881{/,file:/alluxio/core/server/common/src/main/webapp/,AVAILABLE}{/alluxio/core/server/common/src/main/webapp}
2018-08-24 18:26:12,336 INFO  ServerConnector - Started ServerConnector@7b139eab{HTTP/1.1}{0.0.0.0:19999}
2018-08-24 18:26:12,336 INFO  Server - Started @25259ms
2018-08-24 18:26:12,336 INFO  WebServer - Alluxio Master Web service started @ /0.0.0.0:19999
2018-08-24 18:26:12,336 INFO  AlluxioMasterProcess - Alluxio master version 1.9.0-SNAPSHOT started. bindHost=/0.0.0.0:19998, connectHost=AlluxioMaster/10.114.10.156:19998, rpcPort=19998, webPort=19999




On Tuesday, August 21, 2018 at 12:58:53 PM UTC-7, [hidden email] wrote:
Hi Clayton,

Can you add these properties to conf/log4j.properties and restart the Alluxio master:

log4j.logger.com.amazonaws=WARN
log4j.logger.com.amazonaws.request=DEBUG
log4j.logger.org.apache.http.wire=DEBUG

Please share logs/master.log after making the change and retrying the mount.

Adit

On Sunday, August 19, 2018 at 4:10:45 PM UTC-7, Clayton McClure wrote:

I'm new to Alluxio, still getting initial setup working.  Any help would be greatly appreciated.

Alluxio Version:  Current clone (1.9 snapshot)
Env:  AWS in us-west-2
UFS: S3

I've followed the guide for getting it setup using Vagrant and have a cluster setup in EC2.  I have an IAM profile and a test s3 bucket with data in it.  I've validated I can read data in the bucket from the AlluxioMaster:

[ec2-user@AlluxioMaster ~]$ aws s3 ls s3://redacted-bucket/
                           PRE some-data/
                           PRE some-other-data/
                           PRE tmp/

I also copied up a test file and back down from that bucket.

However, when I try to mount the s3 bucket I get a 400 Bad Request:

ec2-user@AlluxioMaster ~]$ /alluxio/bin/alluxio fs mount --readonly alluxio://localhost:19998/mnt/s3 s3a://redacted-bucket/
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 11E74C12CC0B92A8; S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=), S3 Extended Request ID: 2PcA2Cm0kJbSNe3pNZgrYbnB6uhjdELPp9XqVm4KK7SL8WJtTU0P1Iq3o/NyxQjNLeEFMMuXfbQ=


I've tried a whole bunch of different s3 configuration options, such as disabling dns, using https, specifying the s3 endpoint, etc, but I continually get the same error.  

Does anyone have a suggestion for how else I might trouble shoot this?

Thanks,

Clayton

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with Mounting S3 (Getting 400 Bad Request)

Calvin Jia
Hi,

Could you share the alluxio properties you are running with? Also, is the bucket you are accessing also in us-west-2?

Thanks,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.