Robin Robin policy does not balance

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Robin Robin policy does not balance

William Callaghan
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Robin Robin policy does not balance

Yupeng Fu
Hi William,

How did you configure the RoundRobin policy? Did you pass the configuration to Spark JAVA_OPTS?

Cheers,


On Tue, Aug 9, 2016 at 9:03 AM, William Callaghan <[hidden email]> wrote:
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Robin Robin policy does not balance

William Callaghan
Configuration is in JAVA_OPTS and in alluxio-site.properties on each node.

On Tuesday, August 9, 2016 at 12:18:57 PM UTC-4, Yupeng Fu wrote:
Hi William,

How did you configure the RoundRobin policy? Did you pass the configuration to Spark JAVA_OPTS?

Cheers,

Yupeng

<a href="http://www.alluxio.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio Inc

On Tue, Aug 9, 2016 at 9:03 AM, William Callaghan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="UZ5So6fOCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">wca...@...> wrote:
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="UZ5So6fOCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Robin Robin policy does not balance

Yupeng Fu
That will affect the server only. 
Spark calls Alluxio's client, and the client uses the policy to decide where to write. So you'll need to pass the configurations to Alluxio's client.

Specifically, you can add the following lines to spark/conf/spark-env.sh:
export SPARK_JAVA_OPTS="
     -Dalluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy
    $SPARK_JAVA_OPTS 
"

Hope this helps,



On Tue, Aug 9, 2016 at 10:09 AM, William Callaghan <[hidden email]> wrote:
Configuration is in JAVA_OPTS and in alluxio-site.properties on each node.

On Tuesday, August 9, 2016 at 12:18:57 PM UTC-4, Yupeng Fu wrote:
Hi William,

How did you configure the RoundRobin policy? Did you pass the configuration to Spark JAVA_OPTS?

Cheers,


On Tue, Aug 9, 2016 at 9:03 AM, William Callaghan <[hidden email]> wrote:
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Robin Robin policy does not balance

Yupeng Fu
Hi,

Were you able to use the policy with the suggested approach?

On Tuesday, August 9, 2016 at 10:41:32 AM UTC-7, Yupeng Fu wrote:
That will affect the server only. 
Spark calls Alluxio's client, and the client uses the policy to decide where to write. So you'll need to pass the configurations to Alluxio's client.

Specifically, you can add the following lines to spark/conf/spark-env.sh:
export SPARK_JAVA_OPTS="
     -Dalluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy
    $SPARK_JAVA_OPTS 
"

Hope this helps,


Yupeng

<a href="http://www.alluxio.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio Inc

On Tue, Aug 9, 2016 at 10:09 AM, William Callaghan <[hidden email]> wrote:
Configuration is in JAVA_OPTS and in alluxio-site.properties on each node.

On Tuesday, August 9, 2016 at 12:18:57 PM UTC-4, Yupeng Fu wrote:
Hi William,

How did you configure the RoundRobin policy? Did you pass the configuration to Spark JAVA_OPTS?

Cheers,

Yupeng

<a href="http://www.alluxio.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.alluxio.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEO-aqdHfZyi6Oxg9lUcWW5v5b4zg&#39;;return true;">Alluxio Inc

On Tue, Aug 9, 2016 at 9:03 AM, William Callaghan <[hidden email]> wrote:
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Robin Robin policy does not balance

Yupeng Fu
Glad to hear you have the problem solved!


On Thu, Aug 11, 2016 at 8:05 AM, William Callaghan <[hidden email]> wrote:
Yup, thanks!

Sent from my iPhone

On Aug 11, 2016, at 10:27 AM, Yupeng Fu <[hidden email]> wrote:

Hi,

Were you able to use the policy with the suggested approach?

On Tuesday, August 9, 2016 at 10:41:32 AM UTC-7, Yupeng Fu wrote:
That will affect the server only. 
Spark calls Alluxio's client, and the client uses the policy to decide where to write. So you'll need to pass the configurations to Alluxio's client.

Specifically, you can add the following lines to spark/conf/spark-env.sh:
export SPARK_JAVA_OPTS="
     -Dalluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy
    $SPARK_JAVA_OPTS 
"

Hope this helps,



On Tue, Aug 9, 2016 at 10:09 AM, William Callaghan <[hidden email]> wrote:
Configuration is in JAVA_OPTS and in alluxio-site.properties on each node.

On Tuesday, August 9, 2016 at 12:18:57 PM UTC-4, Yupeng Fu wrote:
Hi William,

How did you configure the RoundRobin policy? Did you pass the configuration to Spark JAVA_OPTS?

Cheers,


On Tue, Aug 9, 2016 at 9:03 AM, William Callaghan <[hidden email]> wrote:
Running Alluxio 1.1.1
3 nodes.

Have a spark dataframe with 3 partitions on one node, however I could have a dataframe with any number of partitions. Want to write each partition to a different node in Alluxio. Currently using the RoundRobin policy and all data still goes to one node.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]om.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.