How to specify queue and job_name when using Alluxio on Yarn?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to specify queue and job_name when using Alluxio on Yarn?

wayasxxx
Hi all,

I am trying  Alluxio on Yarn according to the doc: https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

Andrew Audibert
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

wayasxxx
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a <a href="https://alluxio.atlassian.net/projects/ALLUXIO" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;">JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found <a href="https://github.com/Alluxio/alluxio/tree/master/integration/yarn/src/main/java/alluxio/yarn" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;">here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="MSqaQKdPAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">waya...@...> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="MSqaQKdPAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

Andrew Audibert
That error suggests that the master either failed to come up, or is running on a host different from localhost. Do you see an alluxio master container running in YARN? If port 19999 is open, try viewing the alluxio UI in the browser at master_address:19999.

On Wed, Sep 26, 2018 at 1:26 AM <[hidden email]> wrote:
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

wayasxxx
I add some logs and check the yarn RM log. It shows the application master is running and trying to allocate containers : ApplicationMaster.requestAndLaunchContainers
But nothing happens after codes come to ContainerAllocator.requestContainers , the RM doesn't receive any request.  and ContainerAllocator.mOutstandingContainerRequestsLatch is always waiting.
It seems Alluxio on Yarn is not widely used?  I can hardly find any materials about it.


在 2018年9月27日星期四 UTC+8上午12:54:23,Andrew Audibert写道:
That error suggests that the master either failed to come up, or is running on a host different from localhost. Do you see an alluxio master container running in YARN? If port 19999 is open, try viewing the alluxio UI in the browser at master_address:19999.

On Wed, Sep 26, 2018 at 1:26 AM <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Bfmh9v6KAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">waya...@...> wrote:
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the <a href="http://alluxio.user.app.id" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;">alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998: Failed to handshake with master localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998: Failed to handshake with master localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a <a href="https://alluxio.atlassian.net/projects/ALLUXIO" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;">JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found <a href="https://github.com/Alluxio/alluxio/tree/master/integration/yarn/src/main/java/alluxio/yarn" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;">here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="Bfmh9v6KAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

Andrew Audibert
For production environments we recommend running Alluxio alongside YARN instead of on top of it. YARN is designed for running short-lived tasks, but Alluxio servers run indefinitely. However, the YARN integration is still tested with each release, and I know users who are running it successfully.

If the ApplicationMaster is stuck trying to launch the master, it's likely that the node hostname used by YARN doesn't match the hostname you're passing to alluxio-yarn.sh. Can you share the output of $ALLUXIO_HOME/logs/framework.log?

On Sat, Sep 29, 2018 at 2:18 AM <[hidden email]> wrote:
I add some logs and check the yarn RM log. It shows the application master is running and trying to allocate containers : ApplicationMaster.requestAndLaunchContainers
But nothing happens after codes come to ContainerAllocator.requestContainers , the RM doesn't receive any request.  and ContainerAllocator.mOutstandingContainerRequestsLatch is always waiting.
It seems Alluxio on Yarn is not widely used?  I can hardly find any materials about it.


在 2018年9月27日星期四 UTC+8上午12:54:23,Andrew Audibert写道:
That error suggests that the master either failed to come up, or is running on a host different from localhost. Do you see an alluxio master container running in YARN? If port 19999 is open, try viewing the alluxio UI in the browser at master_address:19999.

On Wed, Sep 26, 2018 at 1:26 AM <[hidden email]> wrote:
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

wayasxxx
Thanks. I will think about running Alluxio independently. 

I have tried several versions of hostname, they all failed. There is no log file in  $ALLUXIO_HOME/logs/. I will try some more.

Here is another question, why I have to specify a hostname for master instead of arranging one by yarn or setting up master together with application master? 

在 2018年10月2日星期二 UTC+8上午2:28:20,Andrew Audibert写道:
For production environments we recommend running Alluxio alongside YARN instead of on top of it. YARN is designed for running short-lived tasks, but Alluxio servers run indefinitely. However, the YARN integration is still tested with each release, and I know users who are running it successfully.

If the ApplicationMaster is stuck trying to launch the master, it's likely that the node hostname used by YARN doesn't match the hostname you're passing to alluxio-yarn.sh. Can you share the output of $ALLUXIO_HOME/logs/framework.log?

On Sat, Sep 29, 2018 at 2:18 AM <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pYG9bL7_AQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">waya...@...> wrote:
I add some logs and check the yarn RM log. It shows the application master is running and trying to allocate containers : ApplicationMaster.requestAndLaunchContainers
But nothing happens after codes come to ContainerAllocator.requestContainers , the RM doesn't receive any request.  and ContainerAllocator.mOutstandingContainerRequestsLatch is always waiting.
It seems Alluxio on Yarn is not widely used?  I can hardly find any materials about it.


在 2018年9月27日星期四 UTC+8上午12:54:23,Andrew Audibert写道:
That error suggests that the master either failed to come up, or is running on a host different from localhost. Do you see an alluxio master container running in YARN? If port 19999 is open, try viewing the alluxio UI in the browser at master_address:19999.

On Wed, Sep 26, 2018 at 1:26 AM <[hidden email]> wrote:
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the <a href="http://alluxio.user.app.id" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.user.app.id\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzzCxPbcvESbygOc98ZBKCEcYE7w&#39;;return true;">alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998: Failed to handshake with master localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998: Failed to handshake with master localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/<a href="http://127.0.0.1:19998" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A19998\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNET2tIqY3dnEJ8PrDZT_KbH_DTxVw&#39;;return true;">127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a <a href="https://alluxio.atlassian.net/projects/ALLUXIO" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Falluxio.atlassian.net%2Fprojects%2FALLUXIO\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHDmldYZp-ylSBi8UXe6FmEsQE9mQ&#39;;return true;">JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found <a href="https://github.com/Alluxio/alluxio/tree/master/integration/yarn/src/main/java/alluxio/yarn" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FAlluxio%2Falluxio%2Ftree%2Fmaster%2Fintegration%2Fyarn%2Fsrc%2Fmain%2Fjava%2Falluxio%2Fyarn\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG06PuSxTHJ5UHGdct7TDyeeRfVsQ&#39;;return true;">here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pYG9bL7_AQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to specify queue and job_name when using Alluxio on Yarn?

Andrew Audibert
The applications that connect to Alluxio need to know where the master is running, so it usually makes sense to specify a fixed address. That way, if you restart the Alluxio-on-YARN cluster, the master will come up in the same place. But if you don't care which machine the master starts on, you can set the master address to "any" and an arbitrary master node will be used.

Do you see anything in the Alluxio application master logs? The application master should send a request to the RM for a container, then wait to be allocated a container. If the application master is stuck on ContainerAllocator.mOutstandingContainerRequestsLatch, the RM must not be able to fulfill the container request sent by the AM, e.g. because there isn't enough memory/cpu available, or because the hostname doesn't match.

On Sat, Oct 6, 2018 at 9:21 PM <[hidden email]> wrote:
Thanks. I will think about running Alluxio independently. 

I have tried several versions of hostname, they all failed. There is no log file in  $ALLUXIO_HOME/logs/. I will try some more.

Here is another question, why I have to specify a hostname for master instead of arranging one by yarn or setting up master together with application master? 

在 2018年10月2日星期二 UTC+8上午2:28:20,Andrew Audibert写道:
For production environments we recommend running Alluxio alongside YARN instead of on top of it. YARN is designed for running short-lived tasks, but Alluxio servers run indefinitely. However, the YARN integration is still tested with each release, and I know users who are running it successfully.

If the ApplicationMaster is stuck trying to launch the master, it's likely that the node hostname used by YARN doesn't match the hostname you're passing to alluxio-yarn.sh. Can you share the output of $ALLUXIO_HOME/logs/framework.log?

On Sat, Sep 29, 2018 at 2:18 AM <[hidden email]> wrote:
I add some logs and check the yarn RM log. It shows the application master is running and trying to allocate containers : ApplicationMaster.requestAndLaunchContainers
But nothing happens after codes come to ContainerAllocator.requestContainers , the RM doesn't receive any request.  and ContainerAllocator.mOutstandingContainerRequestsLatch is always waiting.
It seems Alluxio on Yarn is not widely used?  I can hardly find any materials about it.


在 2018年9月27日星期四 UTC+8上午12:54:23,Andrew Audibert写道:
That error suggests that the master either failed to come up, or is running on a host different from localhost. Do you see an alluxio master container running in YARN? If port 19999 is open, try viewing the alluxio UI in the browser at master_address:19999.

On Wed, Sep 26, 2018 at 1:26 AM <[hidden email]> wrote:
Hi Andrew,

Thanks for your attention. 
I found I can set the job_name and queue_name by slightly changing the running scripts. But the cluster_name is still not available.
I will open a jira soon.

However, I face another problem. After submitting an alluxio job on yarn., it doesn't start to work correctly.
I use alluxio-1.8.0 + hadoop-2.6.5. And I got these errors when I try {${ALLUXIO_HOME}/bin/alluxio runTests}:

$ ./alluxio runTests
2018-09-26 15:35:38,785 INFO  MetricsSystem - Starting sinks with config: {}.
2018-09-26 15:35:38,822 INFO  FileSystemContext - Created filesystem context with id app-2350705014292587112. This ID will be used for identifying info from the client, such as metrics. It can be set manually through the alluxio.user.app.id property
2018-09-26 15:35:38,916 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998
2018-09-26 15:35:38,943 WARN  AbstractClient - Failed to handshake (1) with MetricsMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:38,943 ERROR ClientMasterSync - Failed to heartbeat to the metrics master: {}
alluxio.exception.status.UnavailableException: Failed to connect to MetricsMasterClient @ localhost/127.0.0.1:19998 after 1 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:325)
at alluxio.client.metrics.MetricsMasterClient.heartbeat(MetricsMasterClient.java:84)
at alluxio.client.metrics.ClientMasterSync.heartbeat(ClientMasterSync.java:63)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-26 15:35:38,944 WARN  AbstractClient - Failed to handshake (1) with FileSystemMasterClient @ localhost/127.0.0.1:19998: Failed to handshake with master localhost/127.0.0.1:19998 to load cluster default configuration values
2018-09-26 15:35:39,003 INFO  AbstractClient - Alluxio client (version 1.8.0) is trying to bootstrap-connect with localhost/127.0.0.1:19998 

Thanks,
Anyang



在 2018年9月26日星期三 UTC+8上午6:46:55,Andrew Audibert写道:
Hi wayasxxx,

Those parameters aren't currently exposed in the Alluxio Yarn integration. Please open a JIRA ticket to request the features you're interested in. We're happy to accept code contributions - all the code for the yarn integration can be found here.

Best,
Andrew

On Tue, Sep 25, 2018 at 2:50 AM <[hidden email]> wrote:
Hi all,

I am trying  Alluxio on Yarn according to the doc: https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html
Three parameters can be set to specify the worker_num, hdfs_path and master_host_name.

How can I specify the Yarn cluster name, queue, job name and maybe some other yarn settings?
I think It is necessary for working in a production environment.

thanks,
Appreciate your help in advance.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.