Alluxio Domain Socket 1.8

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Alluxio Domain Socket 1.8

Jais Sebastian
Hello,

I am using Spark.2.2.0 and Alluxio 1.8. Both Spark and Mesos deployed in Mesos framework. I enabled domain socket and short circuit read. Still, I don't see short circuit read in the Alluxio matrix UI. I also noticed EpollSocketChannel and EpollDomainSocketChannel in the logs. Not sure which one is working. Please advise if I am missing any configuration?

18/10/07 18:42:24 INFO HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/07 18:42:24 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to bootstrap-connect with /<ALLUXIO MASTER>:31398
18/10/07 18:42:25 WARN AbstractClient: Alluxio client version (1.8.1) does not match Alluxio cluster version (1.8.1-SNAPSHOT)
18/10/07 18:42:25 INFO AbstractClient: Alluxio client has bootstrap-connected with <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO FileSystemContext: Created filesystem context with id app-4009908646368875585. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=<ALLUXIO WORKER - CLIENT LOCATION>, rack=null)
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with BlockMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with BlockMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO NettyUtils: EPOLL_MODE is available
18/10/07 18:42:25 INFO NettyUtils: EPOLL_MODE is available
18/10/07 18:42:25 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@2857d382, remoteAddress: /<ALLUXIO WORKER1>:31499).
18/10/07 18:42:25 WARN Bootstrap: Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0x70a46f99]'
18/10/07 18:42:25 WARN Bootstrap: Unknown channel option 'TCP_NODELAY' for channel '[id: 0x70a46f99]'
18/10/07 18:42:25 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollDomainSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@495148cf, remoteAddress: /tmp/domain/d3).
18/10/07 18:42:25 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2148 bytes result sent to driver
18/10/07 18:42:29 INFO CoarseGrainedExecutorBackend: Got assigned task 1
18/10/07 18:42:29 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
18/10/07 18:42:29 INFO TorrentBroadcast: Started reading broadcast variable 2
18/10/07 18:42:29 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 18:42:29 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.0 KB, free 1663.2 MB)
18/10/07 18:42:29 INFO TorrentBroadcast: Reading broadcast variable 2 took 26 ms
18/10/07 18:42:29 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 19.3 KB, free 1663.2 MB)
18/10/07 18:42:30 INFO CodeGenerator: Code generated in 233.625139 ms
18/10/07 18:42:30 INFO FileScanRDD: Reading File path: alluxio://<ALLUXIO MASTER>:31398/<ALLUXIO PATH>/jQBW1movies001.parquet/part-00000-ab022e3e-1d19-4766-a0d5-21385e26e834-c000.snappy.parquet, range: 0-27163, partition values: [empty row]
18/10/07 18:42:30 INFO TorrentBroadcast: Started reading broadcast variable 1
18/10/07 18:42:30 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 18:42:30 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 26.8 KB, free 1663.1 MB)
18/10/07 18:42:30 INFO TorrentBroadcast: Reading broadcast variable 1 took 24 ms
18/10/07 18:42:30 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 401.4 KB, free 1662.8 MB)
18/10/07 18:42:30 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@4f55511e, remoteAddress: /<ALLUXIO WORKER2>:31499).
18/10/07 18:42:30 INFO CodecPool: Got brand-new decompressor [.snappy]
18/10/07 18:42:30 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 86362 bytes result sent to driver
18/10/07 18:48:44 INFO CoarseGrainedExecutorBackend: Got assigned task 2
18/10/07 18:48:44 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
18/10/07 18:48:44 INFO TorrentBroadcast: Started reading broadcast variable 3
18/10/07 18:48:44 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 27.4 KB, free 1662.7 MB)
18/10/07 18:48:44 INFO TorrentBroadcast: Reading broadcast variable 3 took 12 ms
18/10/07 18:48:44 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 75.8 KB, free 1662.7 MB)
18/10/07 18:48:45 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2105 bytes result sent to driver
....
....
18/10/07 18:58:25 INFO DynamicResourcePool: Resource [id: 0x9763741e, L:null - R:/tmp/domain/d3] is garbage collected.
18/10/07 18:58:25 INFO NettyChannelPool: Channel closed
18/10/07 19:05:30 INFO CoarseGrainedExecutorBackend: Got assigned task 16
18/10/07 19:05:30 INFO Executor: Running task 0.0 in stage 16.0 (TID 16)
18/10/07 19:05:30 INFO TorrentBroadcast: Started reading broadcast variable 25
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 7.0 KB, free 1663.2 MB)
18/10/07 19:05:30 INFO TorrentBroadcast: Reading broadcast variable 25 took 9 ms
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 19.3 KB, free 1663.2 MB)
18/10/07 19:05:30 INFO FileScanRDD: Reading File path: alluxio://<ALLUXIO MASTER>:31398/<ALLUXIO PATH>/part-00000-2fa073df-653d-4dfd-81fb-79a5f3a105a1-c000.snappy.parquet, range: 0-27163, partition values: [empty row]
18/10/07 19:05:30 INFO TorrentBroadcast: Started reading broadcast variable 24
18/10/07 19:05:30 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 26.8 KB, free 1663.1 MB)
18/10/07 19:05:30 INFO TorrentBroadcast: Reading broadcast variable 24 took 13 ms
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 401.4 KB, free 1662.8 MB)
18/10/07 19:05:30 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 86319 bytes result sent to driver

Regards,
Jais

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Alluxio Domain Socket 1.8

adit
Jais, were you able to resolve the issue?

On Sunday, October 7, 2018 at 12:31:58 PM UTC-7, Jais Sebastian wrote:
Hello,

I am using Spark.2.2.0 and Alluxio 1.8. Both Spark and Mesos deployed in Mesos framework. I enabled domain socket and short circuit read. Still, I don't see short circuit read in the Alluxio matrix UI. I also noticed EpollSocketChannel and EpollDomainSocketChannel in the logs. Not sure which one is working. Please advise if I am missing any configuration?

18/10/07 18:42:24 INFO HeartbeatThread: Hearbeat Master Metrics Sync is interrupted.
18/10/07 18:42:24 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to bootstrap-connect with /<ALLUXIO MASTER>:31398
18/10/07 18:42:25 WARN AbstractClient: Alluxio client version (1.8.1) does not match Alluxio cluster version (1.8.1-SNAPSHOT)
18/10/07 18:42:25 INFO AbstractClient: Alluxio client has bootstrap-connected with <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO FileSystemContext: Created filesystem context with id app-4009908646368875585. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the <a href="http://alluxio.user.app.id" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;">alluxio.user.app.id property
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with MetricsMasterClient @ /<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with MetricsMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=<ALLUXIO WORKER - CLIENT LOCATION>, rack=null)
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with FileSystemMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Alluxio client (version 1.8.1) is trying to connect with BlockMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO AbstractClient: Client registered with BlockMasterClient @ <ALLUXIO MASTER>/<ALLUXIO MASTER>:31398
18/10/07 18:42:25 INFO NettyUtils: EPOLL_MODE is available
18/10/07 18:42:25 INFO NettyUtils: EPOLL_MODE is available
18/10/07 18:42:25 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@2857d382, remoteAddress: /<ALLUXIO WORKER1>:31499).
18/10/07 18:42:25 WARN Bootstrap: Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0x70a46f99]'
18/10/07 18:42:25 WARN Bootstrap: Unknown channel option 'TCP_NODELAY' for channel '[id: 0x70a46f99]'
18/10/07 18:42:25 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollDomainSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@495148cf, remoteAddress: /tmp/domain/d3).
18/10/07 18:42:25 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2148 bytes result sent to driver
18/10/07 18:42:29 INFO CoarseGrainedExecutorBackend: Got assigned task 1
18/10/07 18:42:29 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
18/10/07 18:42:29 INFO TorrentBroadcast: Started reading broadcast variable 2
18/10/07 18:42:29 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 18:42:29 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.0 KB, free 1663.2 MB)
18/10/07 18:42:29 INFO TorrentBroadcast: Reading broadcast variable 2 took 26 ms
18/10/07 18:42:29 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 19.3 KB, free 1663.2 MB)
18/10/07 18:42:30 INFO CodeGenerator: Code generated in 233.625139 ms
18/10/07 18:42:30 INFO FileScanRDD: Reading File path: alluxio://<ALLUXIO MASTER>:31398/<ALLUXIO PATH>/jQBW1movies001.parquet/part-00000-ab022e3e-1d19-4766-a0d5-21385e26e834-c000.snappy.parquet, range: 0-27163, partition values: [empty row]
18/10/07 18:42:30 INFO TorrentBroadcast: Started reading broadcast variable 1
18/10/07 18:42:30 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 18:42:30 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 26.8 KB, free 1663.1 MB)
18/10/07 18:42:30 INFO TorrentBroadcast: Reading broadcast variable 1 took 24 ms
18/10/07 18:42:30 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 401.4 KB, free 1662.8 MB)
18/10/07 18:42:30 INFO NettyChannelPool: Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true), EPOLL_MODE=LEVEL_TRIGGERED}, handler: alluxio.network.netty.NettyClient$1@4f55511e, remoteAddress: /<ALLUXIO WORKER2>:31499).
18/10/07 18:42:30 INFO CodecPool: Got brand-new decompressor [.snappy]
18/10/07 18:42:30 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 86362 bytes result sent to driver
18/10/07 18:48:44 INFO CoarseGrainedExecutorBackend: Got assigned task 2
18/10/07 18:48:44 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
18/10/07 18:48:44 INFO TorrentBroadcast: Started reading broadcast variable 3
18/10/07 18:48:44 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 27.4 KB, free 1662.7 MB)
18/10/07 18:48:44 INFO TorrentBroadcast: Reading broadcast variable 3 took 12 ms
18/10/07 18:48:44 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 75.8 KB, free 1662.7 MB)
18/10/07 18:48:45 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2105 bytes result sent to driver
....
....
18/10/07 18:58:25 INFO DynamicResourcePool: Resource [id: 0x9763741e, L:null - R:/tmp/domain/d3] is garbage collected.
18/10/07 18:58:25 INFO NettyChannelPool: Channel closed
18/10/07 19:05:30 INFO CoarseGrainedExecutorBackend: Got assigned task 16
18/10/07 19:05:30 INFO Executor: Running task 0.0 in stage 16.0 (TID 16)
18/10/07 19:05:30 INFO TorrentBroadcast: Started reading broadcast variable 25
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 7.0 KB, free 1663.2 MB)
18/10/07 19:05:30 INFO TorrentBroadcast: Reading broadcast variable 25 took 9 ms
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 19.3 KB, free 1663.2 MB)
18/10/07 19:05:30 INFO FileScanRDD: Reading File path: alluxio://<ALLUXIO MASTER>:31398/<ALLUXIO PATH>/part-00000-2fa073df-653d-4dfd-81fb-79a5f3a105a1-c000.snappy.parquet, range: 0-27163, partition values: [empty row]
18/10/07 19:05:30 INFO TorrentBroadcast: Started reading broadcast variable 24
18/10/07 19:05:30 INFO TransportClientFactory: Successfully created connection to /<SPARK MASTER>:42341 after 1 ms (0 ms spent in bootstraps)
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 26.8 KB, free 1663.1 MB)
18/10/07 19:05:30 INFO TorrentBroadcast: Reading broadcast variable 24 took 13 ms
18/10/07 19:05:30 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 401.4 KB, free 1662.8 MB)
18/10/07 19:05:30 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 86319 bytes result sent to driver

Regards,
Jais

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.