Strange Alluxio blocks behavior in 1.7x release

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Strange Alluxio blocks behavior in 1.7x release

Thai Bui
I've been using Alluxio 1.7.0 build for awhile and have observed a few block corruptions / metadata problem. I will try to upgrade to 1.7.1 and see if things will be improved but just wanted to ask here anyway to see if any users see the same thing:

1. Hive reading Parquet expect magic number at tail problem. To fix this, I had to unmount/mount the bucket and read the data again. The error will look like this:

Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: alluxio://x.x.x.x:19998/mnt/some/path/part-r-00001.snappy.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [8, 50, 97, 52]

I have seen this problem a lot in 1.5x release, had to turn off Alluxio, read directly from S3. After upgrading to 1.7x release, I've never seen this problem again until today. Our set up is Hive on Tez, reading data from S3 + Alluxio. Alluxio is used as a read-only mount point and the data is immutable (both in Alluxio and S3). Given the immutability nature of the data, I wonder how could such a block corruption occur? Our Linux filesystem is ZFS with block checksum and automatic corrupted block prepare enabled so this must be a problem that happens when Alluxio is writing the block incorrectly (the output stream is not closed properly? prematurely?)

More error stack trace here:

2018-06-07T12:16:53,398 ERROR [HiveServer2-Background-Pool: Thread-4310] ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1528322657674_0013_5_00, diagnostics=[Task failed, taskId=task_1528322657674_0013_5_00_000047, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1528322657674_0013_5_00_000047_0:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: alluxio://x.x.x.x:19998/mnt/x/part-r-00001.snappy.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [8, 50, 97, 52] at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: alluxio://x.x.x.x:19998/mnt/x/part-r-00001.snappy.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [8, 50, 97, 52] at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)

2. Block metadata inconsistency problem. This problem is new in 1.7x release. To fix this problem, I had to manually load the block from S3 to Alluxio (via the CLI). After that, the block's metadata is correct again.

2018-06-07T00:54:21,252 ERROR [HiveServer2-Background-Pool: Thread-849] ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1528322657674_0003_9_00, diagnostics=[Task failed, taskId=task_1528322657674_0003_9_00_001163, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1528322657674_0003_9_00_001163_0:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: Block 570509230080 is expected to be 130272532 bytes, but only 130272524 bytes are available. Please ensure its metadata is consistent between Alluxio and UFS. at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: Block 570509230080 is expected to be 130272532 bytes, but only 130272524 bytes are available. Please ensure its metadata is consistent between Alluxio and UFS. at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:145) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111) at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157) at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83) at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703) at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662) at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150) at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Calvin Jia
Hi,

Thanks for reporting these issues!

Are the two errors related or independent?

For the first error, does this only happen when using Tez (not sure if you have another compute framework you can use to just attempt reading the corrupted parquet file)?
You can also take a look at the file in Alluxio and compare against the original in S3 (ie. checksum or just look at the tail bytes). The difference (or the lack of a difference) could help isolate the issue.
For the second error, could you take a look at the on-disk block (ie. /mnt/ramdisk/alluxioworker/570509230080) and see if the size is accurate? If it is not, could you take a look at the worker logs and see if there were any issues reported when loading the file?

Thanks,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Thai Bui
The 1st problem happened again today and so far these have been the behaviors:

* When it first occurred, I would tail the last 32 bytes in Alluxio and compare it with the same file in S3. The bytes looked the same, for example:

/opt/alluxio/bin/alluxio fs tail -c 32 /mnt/some/path/part-r-00000.snappy.parquet | od -t x1
0000000 38 37 31 65 32 64 66 65 30 39 64 35 34 63 30 33
0000020 66 36 61 30 65 38 29 00 0a 31 00 00 50 41 52 31
0000040

* Hive on Tez is configured to tolerate up to 3 runtime tasks failures, and the second retry (reading the same file in Alluxio) works with no error. This has made my effort to tail the last bytes *just* before the problem occurred harder.

On Thu, Jun 7, 2018 at 2:56 PM Calvin Jia <[hidden email]> wrote:
Hi,

Thanks for reporting these issues!

Are the two errors related or independent?

For the first error, does this only happen when using Tez (not sure if you have another compute framework you can use to just attempt reading the corrupted parquet file)?
You can also take a look at the file in Alluxio and compare against the original in S3 (ie. checksum or just look at the tail bytes). The difference (or the lack of a difference) could help isolate the issue.
For the second error, could you take a look at the on-disk block (ie. /mnt/ramdisk/alluxioworker/570509230080) and see if the size is accurate? If it is not, could you take a look at the worker logs and see if there were any issues reported when loading the file?

Thanks,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.


--
Thai

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Calvin Jia
Hey Thai,

Thanks for the additional information. Could you provide the client log of the task that experiences the error? I think the data itself is not corrupted in Alluxio, however the client reading might be used in a way which we did not intend for. For example, if multiple threads are using the same FileInStream object. Which version of Tez are you using and which reader are you using? I can take a look at how they are using the Hadoop client and see if it breaks any assumptions we made.

Cheers,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Thai Bui
Hi Calvin -- Really appreciate you looking into this. I will try to debug the entire flow this weekend as well so see if I can reliably reproduce this. So far this happens very sparsely so it's difficult to debug.

The deployment is Hadoop 3.1 on Hive 3.x on Tez 0.9.1 build. The code is available in these branches


I also have my own Alluxio branch forked from the official branch 1.7 since April. So it's actually on Alluxio 1.7.2-SNAPSHOT. I didn't change any code but just excluded a few dependencies to make Alluxio jar compatible with Hive binaries in my cluster. 


The Hive/LLAP container log is attached. The Parquet library version is the official 1.9.0, the error triggered from this line https://github.com/apache/parquet-mr/blob/apache-parquet-1.9.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L492

Thanks,
Thai

On Fri, Jun 8, 2018 at 3:09 PM Calvin Jia <[hidden email]> wrote:
Hey Thai,

Thanks for the additional information. Could you provide the client log of the task that experiences the error? I think the data itself is not corrupted in Alluxio, however the client reading might be used in a way which we did not intend for. For example, if multiple threads are using the same FileInStream object. Which version of Tez are you using and which reader are you using? I can take a look at how they are using the Hadoop client and see if it breaks any assumptions we made.

Cheers,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.


--
Thai

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

hive-llap-tez-error-20180608.log (39K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Thai Bui
Hi Calvin,

For the 1st problem, I was able to find out a Parquet block with different tail content in Alluxio vs. S3 this time today. The content of the block is correct at the worker's node, but reading via the master will returns incorrect result. I think it's highly likely that this bug occurred at the time of writing the block.

In Alluxio:

/opt/alluxio/bin/alluxio fs tail -c 32 /mnt/some/path/part-r-00000.snappy.parquet | od -t x1
0000000 0c 34 39 64 39 0e a6 17 00 34 0e 86 21 00 32 1a
0000020 c0 21 04 66 63 0e a6 17 18 35 33 32 38 30 39 38
0000040

In S3 downloaded locally:

tail -c -32 part-r-00000.snappy.parquet | od -t x1
0000000    38  37  31  65  32  64  66  65  30  39  64  35  34  63  30  33
0000020    66  36  61  30  65  38  29  00  24  26  00  00  50  41  52  31
0000040

Interestingly, when I dig into the actual block located on the worker node, the block tail content is actually fine. 

In Alluxio's worker node's local file system

tail -c 32 /alluxio/fs/alluxioworker/331031248896 | od -t x1
0000000 38 37 31 65 32 64 66 65 30 39 64 35 34 63 30 33
0000020 66 36 61 30 65 38 29 00 c1 31 00 00 50 41 52 31
0000040

Because of this, free/load the file manually doesn't work, I had to clear the block metadata by unmounting/mounting the bucket from S3 to Alluxio over again.

More info:

check consistency is fine:

/opt/alluxio/bin/alluxio fs checkConsistency /mnt/some/path/part-r-00000.snappy.parquet
/mnt/s3__magpie-parquet/production/feature/2018/06/03/02/part-r-00000.snappy.parquet is consistent with the under storage system.

stat of the block revealed the same stats as the file on s3

/opt/alluxio/bin/alluxio fs stat /mnt/some/path/part-r-00000.snappy.parquet
/mnt/some/path/part-r-00000.snappy.parquet is a file path.
FileInfo{fileId=331048026111, name=part-r-00000.snappy.parquet, path=/mnt/some/path/part-r-00000.snappy.parquet, ufsPath=s3a://some/path/part-r-00000.snappy.parquet, length=39770748, blockSizeBytes=268435456, creationTimeMs=1528328596686, completed=true, folder=false, pinned=false, cacheable=true, persisted=true, blockIds=[331031248896], inMemoryPercentage=0, lastModificationTimesMs=1528328596686, ttl=-1, ttlAction=DELETE, owner=, group=, mode=455, persistenceState=PERSISTED, mountPoint=false, fileBlockInfos=[FileBlockInfo{blockInfo=BlockInfo{id=331031248896, length=39770748, locations=[BlockLocation{workerId=8454988502107513746, address=WorkerNetAddress{host=ip-x-x-x-x, rpcPort=29998, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=ip-x-x-x-x, rack=null)}, tierAlias=SSD}]}, offset=0, ufsLocations=[]}], mountId=1473863132723518797, inAlluxioPercentage=100, ufsFingerprint=TYPE:FILE UFS:s3 OWNER:_ GROUP:_ MODE:448 CONTENT_HASH:e9b283573189f8e9cc48e62d61bc5337 }
Containing the following blocks:
BlockInfo{id=331031248896, length=39770748, locations=[BlockLocation{workerId=8454988502107513746, address=WorkerNetAddress{host=ip-x-x-x-x, rpcPort=29998, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=ip-x-x-x-x, rack=null)}, tierAlias=SSD}]}

stat of the file on s3

aws s3 ls s3://some/path/part-r-00000.snappy.parquet
2018-06-02 22:21:22   39770748 part-r-00000.snappy.parquet

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

Calvin Jia
Hi Thai,

Thanks for the detailed information.

Reading with the Alluxio command line seems to have completely different data. This could be due to the block being written incorrectly. Is the incorrect data here consistent with the error you see in Spark (ie. expected but found [#, #, #, #])?

The data in the Alluxio worker block looks more similar, but is also not the same as the data from S3 (and does not have a legit PAR1 footer):
0000020 66  36  61  30  65  38  29  00  24  26  00  00  50  41  52  31
0000020 66  36  61  30  65  38  29  00  c1  31  00  00  50  41  52  31

Since these are parquet files, I imagine that there is no client caching happening, because clients do not read the entire file sequentially. The caching (block write) will happen through the Alluxio worker, which should be using a thread safe implementation. Would it be possible to provide a worker log of a worker with a problematic block? Alluxio 1.7.1 does make performance improvements on async caching, but not correctness fixes. 

As a workaround to mounting and unmounting the entire bucket, you can use "bin/alluxio fs rm --alluxioOnly /filepath" to delete the metadata from only Alluxio, then running an ls on it will refetch the metadata from S3 and you can have a shot at a clean load.

Hope this helps and thanks again for the feedback,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Strange Alluxio blocks behavior in 1.7x release

binfan
Administrator
In reply to this post by Thai Bui
Hi Thai,

Just for a reference,  I have seen a similar data corruption issue before from an Alluxio user and it turns out the harddisk was malfunctioning.
In your case, 
the fact that "/opt/alluxio/bin/alluxio fs tail -c 32 /mnt/some/path/part-r-00000.snappy.parquet | od -t x1" 
returns different result from 
"tail -c 32 /alluxio/fs/alluxioworker/331031248896 | od -t x1"
really makes me to suggest to rule out this possibility.

Best,

- Bin

On Tuesday, June 12, 2018 at 1:59:15 PM UTC-7, Thai Bui wrote:
Hi Calvin,

For the 1st problem, I was able to find out a Parquet block with different tail content in Alluxio vs. S3 this time today. The content of the block is correct at the worker's node, but reading via the master will returns incorrect result. I think it's highly likely that this bug occurred at the time of writing the block.

In Alluxio:

/opt/alluxio/bin/alluxio fs tail -c 32 /mnt/some/path/part-r-00000.snappy.parquet | od -t x1
0000000 0c 34 39 64 39 0e a6 17 00 34 0e 86 21 00 32 1a
0000020 c0 21 04 66 63 0e a6 17 18 35 33 32 38 30 39 38
0000040

In S3 downloaded locally:

tail -c -32 part-r-00000.snappy.parquet | od -t x1
0000000    38  37  31  65  32  64  66  65  30  39  64  35  34  63  30  33
0000020    66  36  61  30  65  38  29  00  24  26  00  00  50  41  52  31
0000040

Interestingly, when I dig into the actual block located on the worker node, the block tail content is actually fine. 

In Alluxio's worker node's local file system

tail -c 32 /alluxio/fs/alluxioworker/331031248896 | od -t x1
0000000 38 37 31 65 32 64 66 65 30 39 64 35 34 63 30 33
0000020 66 36 61 30 65 38 29 00 c1 31 00 00 50 41 52 31
0000040

Because of this, free/load the file manually doesn't work, I had to clear the block metadata by unmounting/mounting the bucket from S3 to Alluxio over again.

More info:

check consistency is fine:

/opt/alluxio/bin/alluxio fs checkConsistency /mnt/some/path/part-r-00000.snappy.parquet
/mnt/s3__magpie-parquet/production/feature/2018/06/03/02/part-r-00000.snappy.parquet is consistent with the under storage system.

stat of the block revealed the same stats as the file on s3

/opt/alluxio/bin/alluxio fs stat /mnt/some/path/part-r-00000.snappy.parquet
/mnt/some/path/part-r-00000.snappy.parquet is a file path.
FileInfo{fileId=331048026111, name=part-r-00000.snappy.parquet, path=/mnt/some/path/part-r-00000.snappy.parquet, ufsPath=s3a://some/path/part-r-00000.snappy.parquet, length=39770748, blockSizeBytes=268435456, creationTimeMs=1528328596686, completed=true, folder=false, pinned=false, cacheable=true, persisted=true, blockIds=[331031248896], inMemoryPercentage=0, lastModificationTimesMs=1528328596686, ttl=-1, ttlAction=DELETE, owner=, group=, mode=455, persistenceState=PERSISTED, mountPoint=false, fileBlockInfos=[FileBlockInfo{blockInfo=BlockInfo{id=331031248896, length=39770748, locations=[BlockLocation{workerId=8454988502107513746, address=WorkerNetAddress{host=ip-x-x-x-x, rpcPort=29998, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=ip-x-x-x-x, rack=null)}, tierAlias=SSD}]}, offset=0, ufsLocations=[]}], mountId=1473863132723518797, inAlluxioPercentage=100, ufsFingerprint=TYPE:FILE UFS:s3 OWNER:_ GROUP:_ MODE:448 CONTENT_HASH:e9b283573189f8e9cc48e62d61bc5337 }
Containing the following blocks:
BlockInfo{id=331031248896, length=39770748, locations=[BlockLocation{workerId=8454988502107513746, address=WorkerNetAddress{host=ip-x-x-x-x, rpcPort=29998, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=ip-x-x-x-x, rack=null)}, tierAlias=SSD}]}

stat of the file on s3

aws s3 ls s3://some/path/part-r-00000.snappy.parquet
2018-06-02 22:21:22   39770748 part-r-00000.snappy.parquet

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.