Errors when attempting to write to S3 underfs

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Errors when attempting to write to S3 underfs

Jack Kosaian
Hi all,

I'm trying to write blocks such that they are saved in my S3 underfs. Here are the relevant configuration parameters I've used:

Write Type: CACHE_THROUGH
In conf/alluxio-env.sh:
- export ALLUXIO_UNDERFS_ADDRESS=s3n://my_s3_bucket_name/test
- Added appropriate keys for Dfs.s3n.awsAccessKeyId and Dfs.s3n.awsSecretAccessKey to ALLUXIO_JAVA_OPTS

On the first write attempt, however, I receive the following error:

java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://my_s3_bucket_name/test/further/path/file_0.alluxio.0x2F45AC960D5DF4A2.tmp
java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId
        at alluxio.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:130)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:105)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:89)
        ...

I've checked my S3 bucket on the AWS console, and the path test/further/ exists under the bucket, but a zero-length file path_$folder$ exists there.

Does anyone have any insight as to what step I might be missing in the configuration process?

Thanks for any help!
- Jack



--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Xiaohui Liu
Hi, Jack

Alluxio complains about "Invalid configuration key fs.s3n.awsAccessKeyId". So the two options probably were not successfully passed to Alluxio. Check the format you used in config, or paste a snippet here if you are not sure.


On Thursday, May 26, 2016 at 1:30:31 PM UTC+8, Jack Kosaian wrote:
Hi all,

I'm trying to write blocks such that they are saved in my S3 underfs. Here are the relevant configuration parameters I've used:

Write Type: CACHE_THROUGH
In conf/alluxio-env.sh:
- export ALLUXIO_UNDERFS_ADDRESS=s3n://my_s3_bucket_name/test
- Added appropriate keys for Dfs.s3n.awsAccessKeyId and Dfs.s3n.awsSecretAccessKey to ALLUXIO_JAVA_OPTS

On the first write attempt, however, I receive the following error:

java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://my_s3_bucket_name/test/further/path/file_0.alluxio.0x2F45AC960D5DF4A2.tmp
java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId
        at alluxio.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:130)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:105)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:89)
        ...

I've checked my S3 bucket on the AWS console, and the path test/further/ exists under the bucket, but a zero-length file path_$folder$ exists there.

Does anyone have any insight as to what step I might be missing in the configuration process?

Thanks for any help!
- Jack



--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Jiří Šimša
Hi Jack,

Could you specify what version of Alluxio are you using and copy and paste your alluxio-env.sh here? Thanks.

Best,

On Thu, May 26, 2016 at 5:30 AM, Xiaohui Liu <[hidden email]> wrote:
Hi, Jack

Alluxio complains about "Invalid configuration key fs.s3n.awsAccessKeyId". So the two options probably were not successfully passed to Alluxio. Check the format you used in config, or paste a snippet here if you are not sure.


On Thursday, May 26, 2016 at 1:30:31 PM UTC+8, Jack Kosaian wrote:
Hi all,

I'm trying to write blocks such that they are saved in my S3 underfs. Here are the relevant configuration parameters I've used:

Write Type: CACHE_THROUGH
In conf/alluxio-env.sh:
- export ALLUXIO_UNDERFS_ADDRESS=s3n://my_s3_bucket_name/test
- Added appropriate keys for Dfs.s3n.awsAccessKeyId and Dfs.s3n.awsSecretAccessKey to ALLUXIO_JAVA_OPTS

On the first write attempt, however, I receive the following error:

java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://my_s3_bucket_name/test/further/path/file_0.alluxio.0x2F45AC960D5DF4A2.tmp
java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId
        at alluxio.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:130)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:105)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:89)
        ...

I've checked my S3 bucket on the AWS console, and the path test/further/ exists under the bucket, but a zero-length file path_$folder$ exists there.

Does anyone have any insight as to what step I might be missing in the configuration process?

Thanks for any help!
- Jack



--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Jiří Šimša

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Calvin Jia
Hi Jack,

Properties modified in `alluxio-env.sh` will only apply to programs launched from the alluxio/bin folder. If you are using something like Spark to access the data, you will need to pass the keys to that process.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Jack Kosaian
Hi Xiaohui, Jiri, and Calvin,

Thank you all for your responses!

I'm using Alluxio version 1.0.1

Here's a copy of conf/alluxio-env.sh (after the commented block):
if [[ $(uname -s) == Darwin ]]; then
  # Assuming Mac OS X
  export JAVA_HOME=${JAVA_HOME:-$(/usr/libexec/java_home)}
  export ALLUXIO_RAM_FOLDER=/Volumes/ramdisk
  export ALLUXIO_JAVA_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="
else
  # Assuming Linux
  if [[ -z "$JAVA_HOME" ]]; then
    if [ -d /usr/lib/jvm/java-7-oracle ]; then
      export JAVA_HOME=/usr/lib/jvm/java-7-oracle
    else
      # openjdk will set this
      if [[ -d /usr/lib/jvm/jre-1.7.0 ]]; then
        export JAVA_HOME=/usr/lib/jvm/jre-1.7.0
      fi
    fi
  fi
  if [[ -z "${ALLUXIO_RAM_FOLDER}" ]]; then
    export ALLUXIO_RAM_FOLDER="/mnt/ramdisk"
  fi
fi

if [[ -z "${JAVA_HOME}" ]]; then
  export JAVA_HOME="$(dirname $(which java))/.."
fi

export JAVA="${JAVA_HOME}/bin/java"
export ALLUXIO_MASTER_ADDRESS=${ALLUXIO_MASTER_ADDRESS:-localhost}
export ALLUXIO_UNDERFS_ADDRESS=s3n://jackstachyonexperiments/tmp
#export ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-hdfs://localhost:9000}
export ALLUXIO_WORKER_MEMORY_SIZE=8GB

export ALLUXIO_SSH_FOREGROUND=${ALLUXIO_SSH_FOREGROUND:-"yes"}
export ALLUXIO_WORKER_SLEEP=${ALLUXIO_WORKER_SLEEP:-"0.02"}

# Prepend Alluxio classes before classes specified by ALLUXIO_CLASSPATH
# in the Java classpath.  May be necessary if there are jar conflicts
#export ALLUXIO_PREPEND_ALLUXIO_CLASSES=${ALLUXIO_PREPEND_ALLUXIO_CLASSES:-"yes"}

# Where log files are stored. $ALLUXIO_HOME/logs by default.
#export ALLUXIO_LOGS_DIR=${ALLUXIO_LOGS_DIR:-${ALLUXIO_HOME}/logs}

CONF_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

export ALLUXIO_JAVA_OPTS+="
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
  -Dfs.s3n.awsAccessKeyId=<AWS_ACCESS_KEY>
    -Dfs.s3n.awsSecretAccessKey=<AWS_SECRET_KEY>
  -Xmx10g
"

# Master specific parameters. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

# Worker specific parameters that will be shared to all workers. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

Where <AWS_ACCESS_KEY> and <AWS_SECRET_KEY> are replaced by my access key and secret key, respectively.

@Calvin: I am using another process to make read/write requests. I'll try adding the relevant key information to their command lines.

Thanks!

On Thursday, May 26, 2016 at 5:39:49 PM UTC-7, Calvin Jia wrote:
Hi Jack,

Properties modified in `alluxio-env.sh` will only apply to programs launched from the alluxio/bin folder. If you are using something like Spark to access the data, you will need to pass the keys to that process.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Jack Kosaian
Calvin,

Your suggestion appears to have worked! Files are being written to s3 and reads from s3 appear to be being processed just fine.

On writes, though there are some warnings that come up. They appear to have the following form:
"WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/tmp%2Fpath/to/file_0.alluxio.0x732798DFCDC9A483.tmp_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Fri, 27 May 2016 03:53:24 GMT, Authorization: AWS <ACCESS_KEY_HERE>:<jibberish>, User-Agent: JetS3t/0.8.1 (Linux/3.14.35-28.38.amzn1.x86_64; amd64; en; JVM 1.7.0_101), Host: jackstachyonexperiments.s3.amazonaws.com], Response Headers: [x-amz-request-id: <jibberish>, x-amz-id-2: <jibberish>, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Fri, 27 May 2016 03:53:24 GMT, Server: AmazonS3]"

Have you seen these warnings before? If so, do you know whether they are an issue? The operations I'm attempting to perform seem to have worked correctly despite these warnings.

Thanks again!

On Thursday, May 26, 2016 at 8:13:41 PM UTC-7, Jack Kosaian wrote:
Hi Xiaohui, Jiri, and Calvin,

Thank you all for your responses!

I'm using Alluxio version 1.0.1

Here's a copy of conf/alluxio-env.sh (after the commented block):
if [[ $(uname -s) == Darwin ]]; then
  # Assuming Mac OS X
  export JAVA_HOME=${JAVA_HOME:-$(/usr/libexec/java_home)}
  export ALLUXIO_RAM_FOLDER=/Volumes/ramdisk
  export ALLUXIO_JAVA_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="
else
  # Assuming Linux
  if [[ -z "$JAVA_HOME" ]]; then
    if [ -d /usr/lib/jvm/java-7-oracle ]; then
      export JAVA_HOME=/usr/lib/jvm/java-7-oracle
    else
      # openjdk will set this
      if [[ -d /usr/lib/jvm/jre-1.7.0 ]]; then
        export JAVA_HOME=/usr/lib/jvm/jre-1.7.0
      fi
    fi
  fi
  if [[ -z "${ALLUXIO_RAM_FOLDER}" ]]; then
    export ALLUXIO_RAM_FOLDER="/mnt/ramdisk"
  fi
fi

if [[ -z "${JAVA_HOME}" ]]; then
  export JAVA_HOME="$(dirname $(which java))/.."
fi

export JAVA="${JAVA_HOME}/bin/java"
export ALLUXIO_MASTER_ADDRESS=${ALLUXIO_MASTER_ADDRESS:-localhost}
export ALLUXIO_UNDERFS_ADDRESS=s3n://jackstachyonexperiments/tmp
#export ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-hdfs://localhost:9000}
export ALLUXIO_WORKER_MEMORY_SIZE=8GB

export ALLUXIO_SSH_FOREGROUND=${ALLUXIO_SSH_FOREGROUND:-"yes"}
export ALLUXIO_WORKER_SLEEP=${ALLUXIO_WORKER_SLEEP:-"0.02"}

# Prepend Alluxio classes before classes specified by ALLUXIO_CLASSPATH
# in the Java classpath.  May be necessary if there are jar conflicts
#export ALLUXIO_PREPEND_ALLUXIO_CLASSES=${ALLUXIO_PREPEND_ALLUXIO_CLASSES:-"yes"}

# Where log files are stored. $ALLUXIO_HOME/logs by default.
#export ALLUXIO_LOGS_DIR=${ALLUXIO_LOGS_DIR:-${ALLUXIO_HOME}/logs}

CONF_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

export ALLUXIO_JAVA_OPTS+="
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
  -Dfs.s3n.awsAccessKeyId=<AWS_ACCESS_KEY>
    -Dfs.s3n.awsSecretAccessKey=<AWS_SECRET_KEY>
  -Xmx10g
"

# Master specific parameters. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

# Worker specific parameters that will be shared to all workers. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

Where <AWS_ACCESS_KEY> and <AWS_SECRET_KEY> are replaced by my access key and secret key, respectively.

@Calvin: I am using another process to make read/write requests. I'll try adding the relevant key information to their command lines.

Thanks!

On Thursday, May 26, 2016 at 5:39:49 PM UTC-7, Calvin Jia wrote:
Hi Jack,

Properties modified in `alluxio-env.sh` will only apply to programs launched from the alluxio/bin folder. If you are using something like Spark to access the data, you will need to pass the keys to that process.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Calvin Jia
Hi Jack,

That error is expected because the implementation for exists tries to HEAD the object (on a nonexistent file, it will show this error).

Hope this helps,
Calvin

On Thursday, May 26, 2016 at 9:31:17 PM UTC-7, Jack Kosaian wrote:
Calvin,

Your suggestion appears to have worked! Files are being written to s3 and reads from s3 appear to be being processed just fine.

On writes, though there are some warnings that come up. They appear to have the following form:
"WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/tmp%2Fpath/to/file_0.alluxio.0x732798DFCDC9A483.tmp_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Fri, 27 May 2016 03:53:24 GMT, Authorization: AWS <ACCESS_KEY_HERE>:<jibberish>, User-Agent: JetS3t/0.8.1 (Linux/3.14.35-28.38.amzn1.x86_64; amd64; en; JVM 1.7.0_101), Host: <a href="http://jackstachyonexperiments.s3.amazonaws.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjackstachyonexperiments.s3.amazonaws.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGDThuEHDxHJTfZFsfEMzWgBXPHog&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjackstachyonexperiments.s3.amazonaws.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGDThuEHDxHJTfZFsfEMzWgBXPHog&#39;;return true;">jackstachyonexperiments.s3.amazonaws.com], Response Headers: [x-amz-request-id: <jibberish>, x-amz-id-2: <jibberish>, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Fri, 27 May 2016 03:53:24 GMT, Server: AmazonS3]"

Have you seen these warnings before? If so, do you know whether they are an issue? The operations I'm attempting to perform seem to have worked correctly despite these warnings.

Thanks again!

On Thursday, May 26, 2016 at 8:13:41 PM UTC-7, Jack Kosaian wrote:
Hi Xiaohui, Jiri, and Calvin,

Thank you all for your responses!

I'm using Alluxio version 1.0.1

Here's a copy of conf/alluxio-env.sh (after the commented block):
if [[ $(uname -s) == Darwin ]]; then
  # Assuming Mac OS X
  export JAVA_HOME=${JAVA_HOME:-$(/usr/libexec/java_home)}
  export ALLUXIO_RAM_FOLDER=/Volumes/ramdisk
  export ALLUXIO_JAVA_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="
else
  # Assuming Linux
  if [[ -z "$JAVA_HOME" ]]; then
    if [ -d /usr/lib/jvm/java-7-oracle ]; then
      export JAVA_HOME=/usr/lib/jvm/java-7-oracle
    else
      # openjdk will set this
      if [[ -d /usr/lib/jvm/jre-1.7.0 ]]; then
        export JAVA_HOME=/usr/lib/jvm/jre-1.7.0
      fi
    fi
  fi
  if [[ -z "${ALLUXIO_RAM_FOLDER}" ]]; then
    export ALLUXIO_RAM_FOLDER="/mnt/ramdisk"
  fi
fi

if [[ -z "${JAVA_HOME}" ]]; then
  export JAVA_HOME="$(dirname $(which java))/.."
fi

export JAVA="${JAVA_HOME}/bin/java"
export ALLUXIO_MASTER_ADDRESS=${ALLUXIO_MASTER_ADDRESS:-localhost}
export ALLUXIO_UNDERFS_ADDRESS=s3n://jackstachyonexperiments/tmp
#export ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-hdfs://localhost:9000}
export ALLUXIO_WORKER_MEMORY_SIZE=8GB

export ALLUXIO_SSH_FOREGROUND=${ALLUXIO_SSH_FOREGROUND:-"yes"}
export ALLUXIO_WORKER_SLEEP=${ALLUXIO_WORKER_SLEEP:-"0.02"}

# Prepend Alluxio classes before classes specified by ALLUXIO_CLASSPATH
# in the Java classpath.  May be necessary if there are jar conflicts
#export ALLUXIO_PREPEND_ALLUXIO_CLASSES=${ALLUXIO_PREPEND_ALLUXIO_CLASSES:-"yes"}

# Where log files are stored. $ALLUXIO_HOME/logs by default.
#export ALLUXIO_LOGS_DIR=${ALLUXIO_LOGS_DIR:-${ALLUXIO_HOME}/logs}

CONF_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

export ALLUXIO_JAVA_OPTS+="
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
  -Dfs.s3n.awsAccessKeyId=<AWS_ACCESS_KEY>
    -Dfs.s3n.awsSecretAccessKey=<AWS_SECRET_KEY>
  -Xmx10g
"

# Master specific parameters. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

# Worker specific parameters that will be shared to all workers. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

Where <AWS_ACCESS_KEY> and <AWS_SECRET_KEY> are replaced by my access key and secret key, respectively.

@Calvin: I am using another process to make read/write requests. I'll try adding the relevant key information to their command lines.

Thanks!

On Thursday, May 26, 2016 at 5:39:49 PM UTC-7, Calvin Jia wrote:
Hi Jack,

Properties modified in `alluxio-env.sh` will only apply to programs launched from the alluxio/bin folder. If you are using something like Spark to access the data, you will need to pass the keys to that process.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Errors when attempting to write to S3 underfs

Jack Kosaian
Got it.

Thanks for your help and fast responses!

On Friday, May 27, 2016 at 1:29:56 PM UTC-7, Calvin Jia wrote:
Hi Jack,

That error is expected because the implementation for exists tries to HEAD the object (on a nonexistent file, it will show this error).

Hope this helps,
Calvin

On Thursday, May 26, 2016 at 9:31:17 PM UTC-7, Jack Kosaian wrote:
Calvin,

Your suggestion appears to have worked! Files are being written to s3 and reads from s3 appear to be being processed just fine.

On writes, though there are some warnings that come up. They appear to have the following form:
"WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/tmp%2Fpath/to/file_0.alluxio.0x732798DFCDC9A483.tmp_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Fri, 27 May 2016 03:53:24 GMT, Authorization: AWS <ACCESS_KEY_HERE>:<jibberish>, User-Agent: JetS3t/0.8.1 (Linux/3.14.35-28.38.amzn1.x86_64; amd64; en; JVM 1.7.0_101), Host: <a href="http://jackstachyonexperiments.s3.amazonaws.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjackstachyonexperiments.s3.amazonaws.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGDThuEHDxHJTfZFsfEMzWgBXPHog&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjackstachyonexperiments.s3.amazonaws.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGDThuEHDxHJTfZFsfEMzWgBXPHog&#39;;return true;">jackstachyonexperiments.s3.amazonaws.com], Response Headers: [x-amz-request-id: <jibberish>, x-amz-id-2: <jibberish>, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Fri, 27 May 2016 03:53:24 GMT, Server: AmazonS3]"

Have you seen these warnings before? If so, do you know whether they are an issue? The operations I'm attempting to perform seem to have worked correctly despite these warnings.

Thanks again!

On Thursday, May 26, 2016 at 8:13:41 PM UTC-7, Jack Kosaian wrote:
Hi Xiaohui, Jiri, and Calvin,

Thank you all for your responses!

I'm using Alluxio version 1.0.1

Here's a copy of conf/alluxio-env.sh (after the commented block):
if [[ $(uname -s) == Darwin ]]; then
  # Assuming Mac OS X
  export JAVA_HOME=${JAVA_HOME:-$(/usr/libexec/java_home)}
  export ALLUXIO_RAM_FOLDER=/Volumes/ramdisk
  export ALLUXIO_JAVA_OPTS="-Djava.security.krb5.realm= -Djava.security.krb5.kdc="
else
  # Assuming Linux
  if [[ -z "$JAVA_HOME" ]]; then
    if [ -d /usr/lib/jvm/java-7-oracle ]; then
      export JAVA_HOME=/usr/lib/jvm/java-7-oracle
    else
      # openjdk will set this
      if [[ -d /usr/lib/jvm/jre-1.7.0 ]]; then
        export JAVA_HOME=/usr/lib/jvm/jre-1.7.0
      fi
    fi
  fi
  if [[ -z "${ALLUXIO_RAM_FOLDER}" ]]; then
    export ALLUXIO_RAM_FOLDER="/mnt/ramdisk"
  fi
fi

if [[ -z "${JAVA_HOME}" ]]; then
  export JAVA_HOME="$(dirname $(which java))/.."
fi

export JAVA="${JAVA_HOME}/bin/java"
export ALLUXIO_MASTER_ADDRESS=${ALLUXIO_MASTER_ADDRESS:-localhost}
export ALLUXIO_UNDERFS_ADDRESS=s3n://jackstachyonexperiments/tmp
#export ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-hdfs://localhost:9000}
export ALLUXIO_WORKER_MEMORY_SIZE=8GB

export ALLUXIO_SSH_FOREGROUND=${ALLUXIO_SSH_FOREGROUND:-"yes"}
export ALLUXIO_WORKER_SLEEP=${ALLUXIO_WORKER_SLEEP:-"0.02"}

# Prepend Alluxio classes before classes specified by ALLUXIO_CLASSPATH
# in the Java classpath.  May be necessary if there are jar conflicts
#export ALLUXIO_PREPEND_ALLUXIO_CLASSES=${ALLUXIO_PREPEND_ALLUXIO_CLASSES:-"yes"}

# Where log files are stored. $ALLUXIO_HOME/logs by default.
#export ALLUXIO_LOGS_DIR=${ALLUXIO_LOGS_DIR:-${ALLUXIO_HOME}/logs}

CONF_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

export ALLUXIO_JAVA_OPTS+="
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
  -Dfs.s3n.awsAccessKeyId=<AWS_ACCESS_KEY>
    -Dfs.s3n.awsSecretAccessKey=<AWS_SECRET_KEY>
  -Xmx10g
"

# Master specific parameters. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

# Worker specific parameters that will be shared to all workers. Default to ALLUXIO_JAVA_OPTS.
export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

Where <AWS_ACCESS_KEY> and <AWS_SECRET_KEY> are replaced by my access key and secret key, respectively.

@Calvin: I am using another process to make read/write requests. I'll try adding the relevant key information to their command lines.

Thanks!

On Thursday, May 26, 2016 at 5:39:49 PM UTC-7, Calvin Jia wrote:
Hi Jack,

Properties modified in `alluxio-env.sh` will only apply to programs launched from the alluxio/bin folder. If you are using something like Spark to access the data, you will need to pass the keys to that process.

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.