Cannot locate default realm when launch alluxio to yarn

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot locate default realm when launch alluxio to yarn

kong.mu.biz

Hi,

I'm trying to launch alluxio in yarn by following this page.
And since we have secured HDFS, I set my kerberos settings by following this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

Andrew Audibert
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <[hidden email]> wrote:

Hi,

I'm trying to launch alluxio in yarn by following this page.
And since we have secured HDFS, I set my kerberos settings by following this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

kong.mu.biz
Hi Andrew,

Thanks for your reply!
I followed your suggestion, it seems to work.
But since I need to put the krb5.conf into conf/ directory, does that mean I need to put my keytab there too and modify the alluxio.master.keytab.file and alluxio.worker.keytab.file in conf/alluxio-site.properties file as ${ALLUXIO_HOME}/conf/xxxx.keytab ?

Thanks in advance!
Mu

On Fri, Aug 17, 2018 at 2:36 AM Andrew Audibert <[hidden email]> wrote:
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <[hidden email]> wrote:

Hi,

I'm trying to launch alluxio in yarn by following this page.
And since we have secured HDFS, I set my kerberos settings by following this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

Andrew Audibert
That sounds right. Is it working?

On Thu, Aug 16, 2018 at 7:58 PM Mu Kong <[hidden email]> wrote:
Hi Andrew,

Thanks for your reply!
I followed your suggestion, it seems to work.
But since I need to put the krb5.conf into conf/ directory, does that mean I need to put my keytab there too and modify the alluxio.master.keytab.file and alluxio.worker.keytab.file in conf/alluxio-site.properties file as ${ALLUXIO_HOME}/conf/xxxx.keytab ?

Thanks in advance!
Mu

On Fri, Aug 17, 2018 at 2:36 AM Andrew Audibert <[hidden email]> wrote:
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <[hidden email]> wrote:

Hi,

I'm trying to launch alluxio in yarn by following this page.
And since we have secured HDFS, I set my kerberos settings by following this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
--

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

kong.mu.biz
Yes, it seems working.
I launched a cluster with 3 workers on yarn and one master outside of yarn.

I think I need to write a client application to test the setup.

On Saturday, August 18, 2018 at 2:18:30 AM UTC+9, Andrew Audibert wrote:
That sounds right. Is it working?

On Thu, Aug 16, 2018 at 7:58 PM Mu Kong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="At2o9bvYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kong....@...> wrote:
Hi Andrew,

Thanks for your reply!
I followed your suggestion, it seems to work.
But since I need to put the krb5.conf into conf/ directory, does that mean I need to put my keytab there too and modify the alluxio.master.keytab.file and alluxio.worker.keytab.file in conf/alluxio-site.properties file as ${ALLUXIO_HOME}/conf/xxxx.keytab ?

Thanks in advance!
Mu

On Fri, Aug 17, 2018 at 2:36 AM Andrew Audibert <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="At2o9bvYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">and...@...> wrote:
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="At2o9bvYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kong....@...> wrote:

Hi,

I'm trying to launch alluxio in yarn by following <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">this page.
And since we have secured HDFS, I set my kerberos settings by following <a href="https://www.alluxio.org/docs/1.7/en/Configuring-Alluxio-with-secure-HDFS.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;">this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="At2o9bvYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="At2o9bvYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

kong.mu.biz
Actually, I found issues related to this.

After I launch the cluster, the workers on the Yarn cluster are complaining about org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

Not sure if I should put them here. Maybe I will start another thread.

On Monday, August 20, 2018 at 11:39:08 AM UTC+9, [hidden email] wrote:
Yes, it seems working.
I launched a cluster with 3 workers on yarn and one master outside of yarn.

I think I need to write a client application to test the setup.

On Saturday, August 18, 2018 at 2:18:30 AM UTC+9, Andrew Audibert wrote:
That sounds right. Is it working?

On Thu, Aug 16, 2018 at 7:58 PM Mu Kong <[hidden email]> wrote:
Hi Andrew,

Thanks for your reply!
I followed your suggestion, it seems to work.
But since I need to put the krb5.conf into conf/ directory, does that mean I need to put my keytab there too and modify the alluxio.master.keytab.file and alluxio.worker.keytab.file in conf/alluxio-site.properties file as ${ALLUXIO_HOME}/conf/xxxx.keytab ?

Thanks in advance!
Mu

On Fri, Aug 17, 2018 at 2:36 AM Andrew Audibert <[hidden email]> wrote:
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <[hidden email]> wrote:

Hi,

I'm trying to launch alluxio in yarn by following <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">this page.
And since we have secured HDFS, I set my kerberos settings by following <a href="https://www.alluxio.org/docs/1.7/en/Configuring-Alluxio-with-secure-HDFS.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;">this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot locate default realm when launch alluxio to yarn

Andrew Audibert
Link to new thread: https://groups.google.com/forum/#!topic/alluxio-users/kj0WqZNpN5U

On Monday, August 20, 2018 at 12:21:13 AM UTC-7, [hidden email] wrote:
Actually, I found issues related to this.

After I launch the cluster, the workers on the Yarn cluster are complaining about org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

Not sure if I should put them here. Maybe I will start another thread.

On Monday, August 20, 2018 at 11:39:08 AM UTC+9, [hidden email] wrote:
Yes, it seems working.
I launched a cluster with 3 workers on yarn and one master outside of yarn.

I think I need to write a client application to test the setup.

On Saturday, August 18, 2018 at 2:18:30 AM UTC+9, Andrew Audibert wrote:
That sounds right. Is it working?

On Thu, Aug 16, 2018 at 7:58 PM Mu Kong <[hidden email]> wrote:
Hi Andrew,

Thanks for your reply!
I followed your suggestion, it seems to work.
But since I need to put the krb5.conf into conf/ directory, does that mean I need to put my keytab there too and modify the alluxio.master.keytab.file and alluxio.worker.keytab.file in conf/alluxio-site.properties file as ${ALLUXIO_HOME}/conf/xxxx.keytab ?

Thanks in advance!
Mu

On Fri, Aug 17, 2018 at 2:36 AM Andrew Audibert <[hidden email]> wrote:
Hi,

Alluxio will copy all contents from it's conf/ directory into the container, so you can add your krb5.conf there. Then in alluxio-env.sh you can point to ${ALLUXIO_HOME}/conf/krb5.conf.

Hope that helps,
Andrew

On Thu, Aug 16, 2018 at 2:58 AM <[hidden email]> wrote:

Hi,

I'm trying to launch alluxio in yarn by following <a href="https://www.alluxio.org/docs/1.8/en/Running-Alluxio-Yarn-Integration.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.8%2Fen%2FRunning-Alluxio-Yarn-Integration.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHbSqh0v7JD6Wf8Wa6ei5I_Rq64_A&#39;;return true;">this page.
And since we have secured HDFS, I set my kerberos settings by following <a href="https://www.alluxio.org/docs/1.7/en/Configuring-Alluxio-with-secure-HDFS.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.alluxio.org%2Fdocs%2F1.7%2Fen%2FConfiguring-Alluxio-with-secure-HDFS.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF9rwXURDw2kXUse7WvNi4ZeKmUig&#39;;return true;">this page.
The only difference is, instead of providing realm and kdc, I have a krb5.conf file on my launch server(client server).

So, I added the following to the conf/alluxio-env.sh:

ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.conf=/home/..../krb5.conf"


After launching the alluxio by running ./integration/yarn/bin/alluxio-yarn.sh,
the application failed very soon.

Got an error in the log:

Launching Application Master
2018-08-16 09:27:51,156 INFO  ApplicationMaster - Starting Application Master with args [-num_workers, 3, -master_address, <master-host>, -resource_path, hdfs:///user/<path-to-the-resource>]
2018-08-16 09:27:51,492 ERROR ApplicationMaster - Error running Application Master
java
.lang.IllegalArgumentException: Can't get Kerberos realm
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
        at alluxio.yarn.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
        ... 3 more
Caused by: KrbException:
Cannot locate default realm
        at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
        ... 9 more


Then I realized that in the container, the krb5.conf will be definitely not there because the alluxio is trying to find that file in the container.

I wonder is there a way that I can use a krb5.conf file in the yarn integration?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Andrew Audibert
<a href="http://alluxio.com/" style="color:rgb(17,85,204);font-size:12.8px" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Falluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEOzcgHeqiDCH9tkk9r99TjTZX7Nw&#39;;return true;">Alluxio, Inc. | <a href="http://bit.ly/alluxio-open-source" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-open-source\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDNVXZleOB7VIXYMM8vGuSeh4NQw&#39;;return true;">Alluxio Open Source | <a href="http://bit.ly/alluxio-get-involved" style="color:rgb(17,85,204)" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fbit.ly%2Falluxio-get-involved\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMkj0A_5qpmy2ZeIJGUV1QLgzxRg&#39;;return true;">Alluxio Community Site

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.