How to solve "Cannot do operations on a closed BlockInStream"?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to solve "Cannot do operations on a closed BlockInStream"?

lazyman322
We run Spark 1.6.1 on Alluxio 1.1.0 and used spark-shell to test RDD cache in Alluxio. But occurred exception about "Cannot do operations on a closed BlockInStream" after we executed  rdd.count(). Can someone give me some suggestions to solve this problem? Thank you.


  1. val rdd = sc.textFile("/path/to/sample-2g")
  2. rdd.persist(StorageLevel.OFF_HEAP)
  3. rdd.count()

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to solve "Cannot do operations on a closed BlockInStream"?

Pei Sun
Spark only supports Tachyon when you uses "OFF_HEAP" persist feature. An Alluxio user implemented an interface for Alluxio [link] but I have never used it.   I recommend you to do this:
var rdd = sc.textFile("/path/to-sample-2g"
rdd = rdd.saveAsTextFile("alluxio://YourAlluxioMaster:port/path")
rdd = sc.textFile("alluxio://YourAlluxioMaster:port/path")

 
Hope this helps.
Pei

On Thu, Jul 14, 2016 at 6:03 AM, lazyman322 <[hidden email]> wrote:
We run Spark 1.6.1 on Alluxio 1.1.0 and used spark-shell to test RDD cache in Alluxio. But occurred exception about "Cannot do operations on a closed BlockInStream" after we executed  rdd.count(). Can someone give me some suggestions to solve this problem? Thank you.


  1. val rdd = sc.textFile("/path/to/sample-2g")
  2. rdd.persist(StorageLevel.OFF_HEAP)
  3. rdd.count()

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to solve "Cannot do operations on a closed BlockInStream"?

Pei Sun
Was your problem resolved?

Pei

On Thu, Jul 14, 2016 at 8:56 AM, Pei Sun <[hidden email]> wrote:
Spark only supports Tachyon when you uses "OFF_HEAP" persist feature. An Alluxio user implemented an interface for Alluxio [link] but I have never used it.   I recommend you to do this:
var rdd = sc.textFile("/path/to-sample-2g"
rdd = rdd.saveAsTextFile("alluxio://YourAlluxioMaster:port/path")
rdd = sc.textFile("alluxio://YourAlluxioMaster:port/path")

 
Hope this helps.
Pei

On Thu, Jul 14, 2016 at 6:03 AM, lazyman322 <[hidden email]> wrote:
We run Spark 1.6.1 on Alluxio 1.1.0 and used spark-shell to test RDD cache in Alluxio. But occurred exception about "Cannot do operations on a closed BlockInStream" after we executed  rdd.count(). Can someone give me some suggestions to solve this problem? Thank you.


  1. val rdd = sc.textFile("/path/to/sample-2g")
  2. rdd.persist(StorageLevel.OFF_HEAP)
  3. rdd.count()

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun



--
Pei Sun

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.