set TTL from spark

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

set TTL from spark

Antonio Si
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Chaomin Yu
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Antonio Si
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciQ-xbeCAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">anton...@...> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ciQ-xbeCAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Cheers,
Chaomin

<a href="http://www.alluxio.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Haoyuan Li
Which version of Alluxio are you running?



On Tue, May 24, 2016 at 6:04 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Antonio Si
Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 6:35:48 PM UTC-7, Haoyuan Li wrote:
Which version of Alluxio are you running?


<a href="http://www.alluxio.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio, Inc. 

On Tue, May 24, 2016 at 6:04 PM, Antonio Si <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="1EH8p0WKAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">anton...@...> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Cheers,
Chaomin

<a href="http://www.alluxio.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="1EH8p0WKAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Antonio Si
Hi,

A followup question. Does the setAttribute only work on a file or I can set it to a directory also?

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 7:48:22 PM UTC-7, Antonio Si wrote:
Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 6:35:48 PM UTC-7, Haoyuan Li wrote:
Which version of Alluxio are you running?


<a href="http://www.alluxio.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio, Inc. 

On Tue, May 24, 2016 at 6:04 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Cheers,
Chaomin

<a href="http://www.alluxio.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Chaomin Yu
Hi Antonio,

setAttribute in general can be applied to a directory. 
But setAttribute for the purpose of TTL only works on a file, because TTL is not yet supported on directories.

For the TTransportException you saw, did you set TTL on a file or a directory?

On Wed, May 25, 2016 at 4:35 PM, Antonio Si <[hidden email]> wrote:
Hi,

A followup question. Does the setAttribute only work on a file or I can set it to a directory also?

Thanks.

Antonio.


On Tuesday, May 24, 2016 at 7:48:22 PM UTC-7, Antonio Si wrote:
Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 6:35:48 PM UTC-7, Haoyuan Li wrote:
Which version of Alluxio are you running?



On Tue, May 24, 2016 at 6:04 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Antonio Si
Thanks Chaomin.

I set the attribute on a directory. I changed it to the files under the directory and it is working fine.

A related question. Let say I set the TTL on a file for 2 hrs. Before it expires, I set it
again for 2 hrs. Does it reset the timer and measure 2 hrs from that point on?

Thanks.

Antonio.

On Wednesday, May 25, 2016 at 4:56:52 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

setAttribute in general can be applied to a directory. 
But setAttribute for the purpose of TTL only works on a file, because TTL is not yet supported on directories.

For the TTransportException you saw, did you set TTL on a file or a directory?

On Wed, May 25, 2016 at 4:35 PM, Antonio Si <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="da44F3TTAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">anton...@...> wrote:
Hi,

A followup question. Does the setAttribute only work on a file or I can set it to a directory also?

Thanks.

Antonio.


On Tuesday, May 24, 2016 at 7:48:22 PM UTC-7, Antonio Si wrote:
Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 6:35:48 PM UTC-7, Haoyuan Li wrote:
Which version of Alluxio are you running?


<a href="http://www.alluxio.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio, Inc. 

On Tue, May 24, 2016 at 6:04 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Cheers,
Chaomin

<a href="http://www.alluxio.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="da44F3TTAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Cheers,
Chaomin

<a href="http://www.alluxio.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: set TTL from spark

Chaomin Yu
Glad to hear that it works for you!

Yes, if you set TTL again, mtime (last modified time) is updated and the new TTL will apply.
You can refer to alluxio.master.file.FileSystemMaster#setAttributeInternal for the code.

Cheers,
Chaomin


On Thu, May 26, 2016 at 9:09 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

I set the attribute on a directory. I changed it to the files under the directory and it is working fine.

A related question. Let say I set the TTL on a file for 2 hrs. Before it expires, I set it
again for 2 hrs. Does it reset the timer and measure 2 hrs from that point on?

Thanks.

Antonio.

On Wednesday, May 25, 2016 at 4:56:52 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

setAttribute in general can be applied to a directory. 
But setAttribute for the purpose of TTL only works on a file, because TTL is not yet supported on directories.

For the TTransportException you saw, did you set TTL on a file or a directory?

On Wed, May 25, 2016 at 4:35 PM, Antonio Si <[hidden email]> wrote:
Hi,

A followup question. Does the setAttribute only work on a file or I can set it to a directory also?

Thanks.

Antonio.


On Tuesday, May 24, 2016 at 7:48:22 PM UTC-7, Antonio Si wrote:
Hi Haoyuan,

I am using Alluxio 1.0.1.

Thanks.

Antonio.

On Tuesday, May 24, 2016 at 6:35:48 PM UTC-7, Haoyuan Li wrote:
Which version of Alluxio are you running?



On Tue, May 24, 2016 at 6:04 PM, Antonio Si <[hidden email]> wrote:
Thanks Chaomin.

When I use setAttribute to change the TTL, I am getting this error:

alluxio.org.apache.thrift.transport.TTransportException
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.FileSystemMasterClientService$Client.recv_setAttribute(FileSystemMasterClientService.java:625)
at alluxio.thrift.FileSystemMasterClientService$Client.setAttribute(FileSystemMasterClientService.java:611)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:352)
at alluxio.client.file.FileSystemMasterClient$15.call(FileSystemMasterClient.java:349)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:349)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:276)

But reading and writing to Alluxio are working fine.

Thanks.

Antonio

On Tuesday, May 24, 2016 at 4:17:22 PM UTC-7, Chaomin Yu wrote:
Hi Antonio,

1. Yes, for an existing file, you can use either use "setTtl shell cmd" or "setAttribute" API to set a TTL:
  •  [setTtl <path> <time to live(in milliseconds)>]      Sets a new TTL value for the file at path.
  •  public void setAttribute(AlluxioURI path, SetAttributeOptions options);  [alluxio.client.file.BaseFileSystem]
2. For now, Alluxio does not support setting TTL as path suffix. You can use either
  • "CreateFileOptions" to set TTL upon file creation, or
  • "SetAttributeOption" after the file has been created.

Hope this helps,
Chaomin

On Tue, May 24, 2016 at 1:10 PM, Antonio Si <[hidden email]> wrote:
Hi,

I have a couple questions regarding setting the TTL of a file:

1.   If a file has already been created, is there a way to modify the metadata of the file to define a TTL for the file?

2.   If I save the file using spark dataframe.write.save("alluxio:// ..."), is there a way I could specify the TTL?

Thanks.

Antonio.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.