Re: alluxio 1.8.0 readlock unreleased

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: alluxio 1.8.0 readlock unreleased

binfan
Administrator
(cc'ing alluxio-users as we are deprecating this alluxio-dev mailing list, see the announcement)

- Bin

On Wed, Sep 26, 2018 at 7:52 PM 王忠民 <[hidden email]> wrote:
Hi,
    I encountered a problem。alluxio-client and alluxio-worker on the same computer. alluxio-client send a BlockOpenRequest to alluxio-worker .

LocalBlockOpenRequest request = LocalBlockOpenRequest.newBuilder().setBlockId(this.mBlockId).setPromote(options.getOptions().getReadType().isPromote()).build(); try { ProtoMessage message = NettyRPC.call(NettyRPCContext.defaults().setChannel(this.mChannel).setTimeout(READ_TIMEOUT_MS), new ProtoMessage(request)); Preconditions.checkState(message.isLocalBlockOpenResponse()); return message.asLocalBlockOpenResponse().getPath(); } catch (Exception var6) { context.releaseNettyChannel(address, this.mChannel); throw var6; }

and then alluxio-worker handle the request.
private void handleBlockOpenRequest(final ChannelHandlerContext ctx,
      final Protocol.LocalBlockOpenRequest request) {
    mRpcExecutor.submit(() ->
        RpcUtils.nettyRPCAndLog(LOG, new RpcUtils.NettyRpcCallable<Void>() {
          @Override
          public Void call() throws Exception {
            if (mLockId == BlockLockManager.INVALID_LOCK_ID) {
              mSessionId = IdUtils.createSessionId();
              // TODO(calvin): Update the locking logic so this can be done better
              if (request.getPromote()) {
                try {
                  mWorker
                      .moveBlock(mSessionId, request.getBlockId(), mStorageTierAssoc.getAlias(0));
                } catch (BlockDoesNotExistException e) {
                  LOG.debug("Block {} to promote does not exist in Alluxio: {}",
                      request.getBlockId(), e.getMessage());
                } catch (Exception e) {
                  LOG.warn("Failed to promote block {}: {}", request.getBlockId(), e.getMessage());
                }
              }

//==========aluxio-client disconnnect at this place ====
              mLockId = mWorker.lockBlock(mSessionId, request.getBlockId());
              mWorker.accessBlock(mSessionId, request.getBlockId());
            } else {
              LOG.warn("Lock block {} without releasing previous block lock {}.",
                  request.getBlockId(), mLockId);
              throw new InvalidWorkerStateException(
                  ExceptionMessage.LOCK_NOT_RELEASED.getMessage(mLockId));
            }
            Protocol.LocalBlockOpenResponse response = Protocol.LocalBlockOpenResponse.newBuilder()
                .setPath(mWorker.readBlock(mSessionId, request.getBlockId(), mLockId)).build();
            ctx.writeAndFlush(new RPCProtoMessage(new ProtoMessage(response)));
            return null;
          }

alluxio-client connection lost when alluxio-worker require readlock . alluxio-worker will deal with the exception.
@Override public void channelUnregistered(ChannelHandlerContext ctx) { if (mLockId != BlockLockManager.INVALID_LOCK_ID) { try { mWorker.unlockBlock(mLockId); } catch (BlockDoesNotExistException e) { LOG.warn("Failed to unlock lock {} with error {}.", mLockId, e.getMessage()); }
mWorker.cleanupSession(mSessionId); } ctx.fireChannelUnregistered(); }

But before mLockId set and channelUnregistered() has already finished, the readlock will be hold all the time .

Is there any other place that frees the readlock?

Thank!



--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.


--
- Bin Fan

Software Engineer
Alluxio
www.alluxio.com


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio 1.8.0 readlock unreleased

wang
Can someone give me a reply?

在 2018年9月28日星期五 UTC+8上午9:15:30,Bin Fan写道:
(cc'ing alluxio-users as we are deprecating this alluxio-dev mailing list, see the <a href="https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;">announcement)

- Bin

On Wed, Sep 26, 2018 at 7:52 PM 王忠民 <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="cjOy9uv0AgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">wangz...@...> wrote:
Hi,
    I encountered a problem。alluxio-client and alluxio-worker on the same computer. alluxio-client send a BlockOpenRequest to alluxio-worker .

LocalBlockOpenRequest request = LocalBlockOpenRequest.newBuilder().setBlockId(this.mBlockId).setPromote(options.getOptions().getReadType().isPromote()).build(); try { ProtoMessage message = NettyRPC.call(NettyRPCContext.defaults().setChannel(this.mChannel).setTimeout(READ_TIMEOUT_MS), new ProtoMessage(request)); Preconditions.checkState(message.isLocalBlockOpenResponse()); return message.asLocalBlockOpenResponse().getPath(); } catch (Exception var6) { context.releaseNettyChannel(address, this.mChannel); throw var6; }

and then alluxio-worker handle the request.
private void handleBlockOpenRequest(final ChannelHandlerContext ctx,
      final Protocol.LocalBlockOpenRequest request) {
    mRpcExecutor.submit(() ->
        RpcUtils.nettyRPCAndLog(LOG, new RpcUtils.NettyRpcCallable<Void>() {
          @Override
          public Void call() throws Exception {
            if (mLockId == BlockLockManager.INVALID_LOCK_ID) {
              mSessionId = IdUtils.createSessionId();
              // TODO(calvin): Update the locking logic so this can be done better
              if (request.getPromote()) {
                try {
                  mWorker
                      .moveBlock(mSessionId, request.getBlockId(), mStorageTierAssoc.getAlias(0));
                } catch (BlockDoesNotExistException e) {
                  LOG.debug("Block {} to promote does not exist in Alluxio: {}",
                      request.getBlockId(), e.getMessage());
                } catch (Exception e) {
                  LOG.warn("Failed to promote block {}: {}", request.getBlockId(), e.getMessage());
                }
              }

//==========aluxio-client disconnnect at this place ====
              mLockId = mWorker.lockBlock(mSessionId, request.getBlockId());
              mWorker.accessBlock(mSessionId, request.getBlockId());
            } else {
              LOG.warn("Lock block {} without releasing previous block lock {}.",
                  request.getBlockId(), mLockId);
              throw new InvalidWorkerStateException(
                  ExceptionMessage.LOCK_NOT_RELEASED.getMessage(mLockId));
            }
            Protocol.LocalBlockOpenResponse response = Protocol.LocalBlockOpenResponse.newBuilder()
                .setPath(mWorker.readBlock(mSessionId, request.getBlockId(), mLockId)).build();
            ctx.writeAndFlush(new RPCProtoMessage(new ProtoMessage(response)));
            return null;
          }

alluxio-client connection lost when alluxio-worker require readlock . alluxio-worker will deal with the exception.
@Override public void channelUnregistered(ChannelHandlerContext ctx) { if (mLockId != BlockLockManager.INVALID_LOCK_ID) { try { mWorker.unlockBlock(mLockId); } catch (BlockDoesNotExistException e) { LOG.warn("Failed to unlock lock {} with error {}.", mLockId, e.getMessage()); }
mWorker.cleanupSession(mSessionId); } ctx.fireChannelUnregistered(); }

But before mLockId set and channelUnregistered() has already finished, the readlock will be hold all the time .

Is there any other place that frees the readlock?

Thank!



--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="cjOy9uv0AgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-dev...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
- Bin Fan

Software Engineer
Alluxio
<a href="http://www.alluxio.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">www.alluxio.com


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio 1.8.0 readlock unreleased

binfan
Administrator
hi Wang,

We actually have code in worker (SessionCleaner) to clean up expired sessions.
Does this help you? Or do you have more issues on the session management?

- Bin

On Wednesday, October 10, 2018 at 8:23:00 PM UTC-7, wang wrote:
Can someone give me a reply?

在 2018年9月28日星期五 UTC+8上午9:15:30,Bin Fan写道:
(cc'ing alluxio-users as we are deprecating this alluxio-dev mailing list, see the <a href="https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;">announcement)

- Bin

On Wed, Sep 26, 2018 at 7:52 PM 王忠民 <[hidden email]> wrote:
Hi,
    I encountered a problem。alluxio-client and alluxio-worker on the same computer. alluxio-client send a BlockOpenRequest to alluxio-worker .

LocalBlockOpenRequest request = LocalBlockOpenRequest.newBuilder().setBlockId(this.mBlockId).setPromote(options.getOptions().getReadType().isPromote()).build(); try { ProtoMessage message = NettyRPC.call(NettyRPCContext.defaults().setChannel(this.mChannel).setTimeout(READ_TIMEOUT_MS), new ProtoMessage(request)); Preconditions.checkState(message.isLocalBlockOpenResponse()); return message.asLocalBlockOpenResponse().getPath(); } catch (Exception var6) { context.releaseNettyChannel(address, this.mChannel); throw var6; }

and then alluxio-worker handle the request.
private void handleBlockOpenRequest(final ChannelHandlerContext ctx,
      final Protocol.LocalBlockOpenRequest request) {
    mRpcExecutor.submit(() ->
        RpcUtils.nettyRPCAndLog(LOG, new RpcUtils.NettyRpcCallable<Void>() {
          @Override
          public Void call() throws Exception {
            if (mLockId == BlockLockManager.INVALID_LOCK_ID) {
              mSessionId = IdUtils.createSessionId();
              // TODO(calvin): Update the locking logic so this can be done better
              if (request.getPromote()) {
                try {
                  mWorker
                      .moveBlock(mSessionId, request.getBlockId(), mStorageTierAssoc.getAlias(0));
                } catch (BlockDoesNotExistException e) {
                  LOG.debug("Block {} to promote does not exist in Alluxio: {}",
                      request.getBlockId(), e.getMessage());
                } catch (Exception e) {
                  LOG.warn("Failed to promote block {}: {}", request.getBlockId(), e.getMessage());
                }
              }

//==========aluxio-client disconnnect at this place ====
              mLockId = mWorker.lockBlock(mSessionId, request.getBlockId());
              mWorker.accessBlock(mSessionId, request.getBlockId());
            } else {
              LOG.warn("Lock block {} without releasing previous block lock {}.",
                  request.getBlockId(), mLockId);
              throw new InvalidWorkerStateException(
                  ExceptionMessage.LOCK_NOT_RELEASED.getMessage(mLockId));
            }
            Protocol.LocalBlockOpenResponse response = Protocol.LocalBlockOpenResponse.newBuilder()
                .setPath(mWorker.readBlock(mSessionId, request.getBlockId(), mLockId)).build();
            ctx.writeAndFlush(new RPCProtoMessage(new ProtoMessage(response)));
            return null;
          }

alluxio-client connection lost when alluxio-worker require readlock . alluxio-worker will deal with the exception.
@Override public void channelUnregistered(ChannelHandlerContext ctx) { if (mLockId != BlockLockManager.INVALID_LOCK_ID) { try { mWorker.unlockBlock(mLockId); } catch (BlockDoesNotExistException e) { LOG.warn("Failed to unlock lock {} with error {}.", mLockId, e.getMessage()); }
mWorker.cleanupSession(mSessionId); } ctx.fireChannelUnregistered(); }

But before mLockId set and channelUnregistered() has already finished, the readlock will be hold all the time .

Is there any other place that frees the readlock?

Thank!



--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
- Bin Fan

Software Engineer
Alluxio
<a href="http://www.alluxio.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">www.alluxio.com


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: alluxio 1.8.0 readlock unreleased

wang
Hi,

   Thanks for your reply. I'm not very familiar with SessionCleaner。But when debug client read , i found client never send a heartbeat message to worker for registering clinet's sessionid to Sessions.


在 2018年10月20日星期六 UTC+8上午8:00:31,Bin Fan写道:
hi Wang,

We actually have code in worker (SessionCleaner) to clean up expired sessions.
Does this help you? Or do you have more issues on the session management?

- Bin

On Wednesday, October 10, 2018 at 8:23:00 PM UTC-7, wang wrote:
Can someone give me a reply?

在 2018年9月28日星期五 UTC+8上午9:15:30,Bin Fan写道:
(cc'ing alluxio-users as we are deprecating this alluxio-dev mailing list, see the <a href="https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/alluxio-dev/WkNSyHu8giQ&#39;;return true;">announcement)

- Bin

On Wed, Sep 26, 2018 at 7:52 PM 王忠民 <[hidden email]> wrote:
Hi,
    I encountered a problem。alluxio-client and alluxio-worker on the same computer. alluxio-client send a BlockOpenRequest to alluxio-worker .

LocalBlockOpenRequest request = LocalBlockOpenRequest.newBuilder().setBlockId(this.mBlockId).setPromote(options.getOptions().getReadType().isPromote()).build(); try { ProtoMessage message = NettyRPC.call(NettyRPCContext.defaults().setChannel(this.mChannel).setTimeout(READ_TIMEOUT_MS), new ProtoMessage(request)); Preconditions.checkState(message.isLocalBlockOpenResponse()); return message.asLocalBlockOpenResponse().getPath(); } catch (Exception var6) { context.releaseNettyChannel(address, this.mChannel); throw var6; }

and then alluxio-worker handle the request.
private void handleBlockOpenRequest(final ChannelHandlerContext ctx,
      final Protocol.LocalBlockOpenRequest request) {
    mRpcExecutor.submit(() ->
        RpcUtils.nettyRPCAndLog(LOG, new RpcUtils.NettyRpcCallable<Void>() {
          @Override
          public Void call() throws Exception {
            if (mLockId == BlockLockManager.INVALID_LOCK_ID) {
              mSessionId = IdUtils.createSessionId();
              // TODO(calvin): Update the locking logic so this can be done better
              if (request.getPromote()) {
                try {
                  mWorker
                      .moveBlock(mSessionId, request.getBlockId(), mStorageTierAssoc.getAlias(0));
                } catch (BlockDoesNotExistException e) {
                  LOG.debug("Block {} to promote does not exist in Alluxio: {}",
                      request.getBlockId(), e.getMessage());
                } catch (Exception e) {
                  LOG.warn("Failed to promote block {}: {}", request.getBlockId(), e.getMessage());
                }
              }

//==========aluxio-client disconnnect at this place ====
              mLockId = mWorker.lockBlock(mSessionId, request.getBlockId());
              mWorker.accessBlock(mSessionId, request.getBlockId());
            } else {
              LOG.warn("Lock block {} without releasing previous block lock {}.",
                  request.getBlockId(), mLockId);
              throw new InvalidWorkerStateException(
                  ExceptionMessage.LOCK_NOT_RELEASED.getMessage(mLockId));
            }
            Protocol.LocalBlockOpenResponse response = Protocol.LocalBlockOpenResponse.newBuilder()
                .setPath(mWorker.readBlock(mSessionId, request.getBlockId(), mLockId)).build();
            ctx.writeAndFlush(new RPCProtoMessage(new ProtoMessage(response)));
            return null;
          }

alluxio-client connection lost when alluxio-worker require readlock . alluxio-worker will deal with the exception.
@Override public void channelUnregistered(ChannelHandlerContext ctx) { if (mLockId != BlockLockManager.INVALID_LOCK_ID) { try { mWorker.unlockBlock(mLockId); } catch (BlockDoesNotExistException e) { LOG.warn("Failed to unlock lock {} with error {}.", mLockId, e.getMessage()); }
mWorker.cleanupSession(mSessionId); } ctx.fireChannelUnregistered(); }

But before mLockId set and channelUnregistered() has already finished, the readlock will be hold all the time .

Is there any other place that frees the readlock?

Thank!



--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
- Bin Fan

Software Engineer
Alluxio
<a href="http://www.alluxio.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEZyav7dNXoLmZvdCMrjvxeKHZdTw&#39;;return true;">www.alluxio.com


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.