Can't delete files recursively from underfs - Minio

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't delete files recursively from underfs - Minio

aputra
Hello, I'm trying to set up Alluxio 1.8.1 on Kubernetes with Minio version=2018-10-18T00:28:58Z.
All tests passes "./alluxio runTests" successfully, but when I'm trying to remove directory recursively with "./alluxio fs rm -R /default_tests_files" it throws an exception. (java.lang.NullPointerException)
I am able to delete files separately with the same command "./alluxio fs -R /default_tests_files/BASIC_NON_BYTE_BUFFER_NO_CACHE_MUST_CACHE".
Also, I can delete whole directory with "--alluxioOnly" - ".alluxio fs rm -R --alluxioOnly /default_tests_files"

I use Apache Spark 2.3.1, and when it tries to remove something recursively I get this error.

Full stack trace:

2018-10-21 23:27:07,582 ERROR FileSystemMasterClientServiceHandler - Exit (Error): Remove: path=/default_tests_files, recursive=true, options=DeleteTOptions(recursive:true, alluxioOnly:false, unchecked:false, commonOptions:FileSystemMasterCommonTOptions(syncIntervalMs:-1))
java.lang.NullPointerException
at alluxio.underfs.s3a.S3AUtils.isUserIdInGrantee(S3AUtils.java:57)
at alluxio.underfs.s3a.S3AUtils.translateBucketAcl(S3AUtils.java:47)
at alluxio.underfs.s3a.S3AUnderFileSystem.getPermissionsInternal(S3AUnderFileSystem.java:523)
at alluxio.underfs.s3a.S3AUnderFileSystem.access$000(S3AUnderFileSystem.java:75)
at alluxio.underfs.s3a.S3AUnderFileSystem$1.get(S3AUnderFileSystem.java:104)
at alluxio.underfs.s3a.S3AUnderFileSystem$1.get(S3AUnderFileSystem.java:101)
at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:125)
at alluxio.underfs.s3a.S3AUnderFileSystem.getPermissions(S3AUnderFileSystem.java:503)
at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:857)
at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:504)
at alluxio.underfs.UnderFileSystemWithLogging$21.call(UnderFileSystemWithLogging.java:404)
at alluxio.underfs.UnderFileSystemWithLogging$21.call(UnderFileSystemWithLogging.java:401)
at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:609)
at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:401)
at alluxio.master.file.UfsSyncChecker.getChildrenInUFS(UfsSyncChecker.java:163)
at alluxio.master.file.UfsSyncChecker.checkDirectory(UfsSyncChecker.java:78)
at alluxio.master.file.SafeUfsDeleter.<init>(SafeUfsDeleter.java:66)
at alluxio.master.file.DefaultFileSystemMaster.deleteInternal(DefaultFileSystemMaster.java:1625)
at alluxio.master.file.DefaultFileSystemMaster.deleteAndJournal(DefaultFileSystemMaster.java:1550)
at alluxio.master.file.DefaultFileSystemMaster.delete(DefaultFileSystemMaster.java:1522)
at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$remove$11(FileSystemMasterClientServiceHandler.java:253)
at alluxio.RpcUtils.call(RpcUtils.java:217)
at alluxio.RpcUtils.call(RpcUtils.java:189)
at alluxio.master.file.FileSystemMasterClientServiceHandler.remove(FileSystemMasterClientServiceHandler.java:245)
at alluxio.thrift.FileSystemMasterClientService$Processor$remove.getResult(FileSystemMasterClientService.java:1669)
at alluxio.thrift.FileSystemMasterClientService$Processor$remove.getResult(FileSystemMasterClientService.java:1653)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at alluxio.master.file.FileSystemMasterClientServiceProcessor.process(FileSystemMasterClientServiceProcessor.java:78)
at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can't delete files recursively from underfs - Minio

Calvin Jia
Hi,

It looks like Minio might not fully support all of S3's APIs. Can you try setting `alluxio.underfs.s3a.inherit_acl=false` in alluxio-site.properties and restarting Alluxio?

Hope this helps,
Calvin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.