Get last element of a DataSe

[Alejandro Alcalde]

Hi,

I am trying to improve the efficiency of this code:

discretized.map(_._2)
          .name("Map V")
          .reduce((_, b) ⇒ b)
          .name("Get Last V")

I am just interested in the last element of discretized.

I've seen this SO question: https://stackoverflow.com/questions/45076310/how-to-get-only-the-last-element-of-the-large-dataset-in-flink

I've tried with this code:

discretized.mapPartition(in ⇒ if (!in.hasNext) in.map(_._2) else Seq())
          .setParallelism(1)
          .name("Get Last V")

But I am getting this error:

Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Best

[Fabian Hueske-2]

Hi,

Can you post the full stacktrace?

Thanks, Fabian

Am Di., 25. Sep. 2018 um 12:55 Uhr schrieb Alejandro Alcalde <[hidden email]>:

Hi,

I am trying to improve the efficiency of this code:

discretized.map(_._2)
          .name("Map V")
          .reduce((_, b) ⇒ b)
          .name("Get Last V")

I am just interested in the last element of discretized.

I've seen this SO question: https://stackoverflow.com/questions/45076310/how-to-get-only-the-last-element-of-the-large-dataset-in-flink

I've tried with this code:

discretized.mapPartition(in ⇒ if (!in.hasNext) in.map(_._2) else Seq())
          .setParallelism(1)
          .name("Get Last V")

But I am getting this error:

Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Best

[Alejandro Alcalde]

Yes, of course

A IDA Discretization
[info]   When computing its discretization
[info]   - Should be computed correctly *** FAILED ***
[info]     org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: The user defined 'open()' method caused an exce
ption: Index: 0, Size: 0                                                                                                            
[info]     at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:623)
[info]     at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:235)
[info]     at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
[info]     at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
[info]     at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:525)
[info]     at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:601)
[info]     at com.elbauldelprogramador.discretizers.IDADiscretizerTransformerSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mc
V$sp$2.apply(IDADiscretizerTransformerSpec.scala:56)                                                                                
[info]     at com.elbauldelprogramador.discretizers.IDADiscretizerTransformerSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mc
V$sp$2.apply(IDADiscretizerTransformerSpec.scala:48)                                                                                
[info]     at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info]     at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]     ...
[info]     Cause: java.lang.Exception: The user defined 'open()' method caused an exception: Index: 0, Size: 0
[info]     at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
[info]     at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
[info]     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
[info]     at java.lang.Thread.run(Thread.java:748)
[info]     ...
[info]     Cause: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
[info]     at java.util.ArrayList.rangeCheck(ArrayList.java:657)
[info]     at java.util.ArrayList.get(ArrayList.java:433)
[info]     at org.apache.flink.ml.package$BroadcastSingleElementMapper.open(package.scala:82)
[info]     at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
[info]     at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494)
[info]     at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
[info]     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
[info]     at java.lang.Thread.run(Thread.java:748)

It seems the problem is not with the code above, but it is caused by it, since cuts is empty.

On Tue, Sep 25, 2018 at 1:33 PM Fabian Hueske <[hidden email]> wrote:

Hi,

Can you post the full stacktrace?

Thanks, Fabian

Am Di., 25. Sep. 2018 um 12:55 Uhr schrieb Alejandro Alcalde <[hidden email]>:

Hi,

I am trying to improve the efficiency of this code:

discretized.map(_._2)
          .name("Map V")
          .reduce((_, b) ⇒ b)
          .name("Get Last V")

I am just interested in the last element of discretized.

I've seen this SO question: https://stackoverflow.com/questions/45076310/how-to-get-only-the-last-element-of-the-large-dataset-in-flink

I've tried with this code:

discretized.mapPartition(in ⇒ if (!in.hasNext) in.map(_._2) else Seq())
          .setParallelism(1)
          .name("Get Last V")

But I am getting this error:

Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Best