Datasets union CompilerException

[Flavio Pompermaier]

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Stephan Ewen]

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Flavio Pompermaier]

Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Fabian Hueske-2]

Hi, is it possible to get a smaller version of that program that reproduces the bug or give a few more details about the structure of the job?

Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <[hidden email]>:

Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Fabian Hueske-2]

Also, can you open a JIRA for the issue? Otherwise it might get lost on the mailing list.

Thanks you!

2015-06-30 10:56 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi, is it possible to get a smaller version of that program that reproduces the bug or give a few more details about the structure of the job?

Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <[hidden email]>:

Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Michele Bertoni]

Hi yesterday on the union I faced an other problem:

at runtime it was saying something like “Union cannot work with dataset of two different types” then it was showing the types and they were exactly the same (Tuple5<Long, String, Long, Long, MyClass>

I solved it changing on field of the tuple from a custom object (MyClass) that was containing only a double to a native double and it worked

I don’t know if it is related but it may be

Michele

Il giorno 30/giu/2015, alle ore 10:57, Fabian Hueske <[hidden email]> ha scritto:

Also, can you open a JIRA for the issue? Otherwise it might get lost on the mailing list.

Thanks you!

2015-06-30 10:56 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi, is it possible to get a smaller version of that program that reproduces the bug or give a few more details about the structure of the job?

Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <[hidden email]>:

Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio

[Stephan Ewen]

I don't think it is related, but another bug...

On Tue, Jun 30, 2015 at 4:44 PM, Michele Bertoni <[hidden email]> wrote:

Hi yesterday on the union I faced an other problem:

at runtime it was saying something like “Union cannot work with dataset of two different types” then it was showing the types and they were exactly the same (Tuple5<Long, String, Long, Long, MyClass>

I solved it changing on field of the tuple from a custom object (MyClass) that was containing only a double to a native double and it worked

I don’t know if it is related but it may be

Michele

Il giorno 30/giu/2015, alle ore 10:57, Fabian Hueske <[hidden email]> ha scritto:

Also, can you open a JIRA for the issue? Otherwise it might get lost on the mailing list.

Thanks you!

2015-06-30 10:56 GMT+02:00 Fabian Hueske <[hidden email]>:

Hi, is it possible to get a smaller version of that program that reproduces the bug or give a few more details about the structure of the job?

Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <[hidden email]>:

Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <[hidden email]>:

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)

at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)

at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)

at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)

at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)

at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)

at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,

Flavio