NPE stacktrace

[Flavio Pompermaier]

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Stephan Ewen]

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Flavio Pompermaier]

The problem is that RuntimeExceptions are not catched by user code so (IMO) Flink should log somewhere all the stacktrace of those exceptions..

At the moment I see only that during a step there was a NullPointerException but I don't know nothing more (no class or line or stacktrace) :(

On Tue, Nov 25, 2014 at 10:24 AM, Stephan Ewen <[hidden email]> wrote:

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Stephan Ewen]

The TaskManager log or the JobManager log or the JobManager WebFrontend show nothing?

On Tue, Nov 25, 2014 at 10:30 AM, Flavio Pompermaier <[hidden email]> wrote:

The problem is that RuntimeExceptions are not catched by user code so (IMO) Flink should log somewhere all the stacktrace of those exceptions..

At the moment I see only that during a step there was a NullPointerException but I don't know nothing more (no class or line or stacktrace) :(

On Tue, Nov 25, 2014 at 10:24 AM, Stephan Ewen <[hidden email]> wrote:

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Flavio Pompermaier]

No..they report just a null pointer in a taskmanager during a certain function

On Tue, Nov 25, 2014 at 10:59 AM, Stephan Ewen <[hidden email]> wrote:

The TaskManager log or the JobManager log or the JobManager WebFrontend show nothing?

On Tue, Nov 25, 2014 at 10:30 AM, Flavio Pompermaier <[hidden email]> wrote:

The problem is that RuntimeExceptions are not catched by user code so (IMO) Flink should log somewhere all the stacktrace of those exceptions..

At the moment I see only that during a step there was a NullPointerException but I don't know nothing more (no class or line or stacktrace) :(

On Tue, Nov 25, 2014 at 10:24 AM, Stephan Ewen <[hidden email]> wrote:

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Stefano Bortoli]

The problem it is that it is only noted the chain of functions, not the exact function. Furthermore, the same code run several time on a smaller dataset (1M records), but fails on this one (9.2M records). Curiously, the CPUs of the cluster were saturated right before the crush (see picture attached).

saluti,
Stefano

2014-11-25 11:15 GMT+01:00 Flavio Pompermaier <[hidden email]>:

No..they report just a null pointer in a taskmanager during a certain function

On Tue, Nov 25, 2014 at 10:59 AM, Stephan Ewen <[hidden email]> wrote:

The TaskManager log or the JobManager log or the JobManager WebFrontend show nothing?

On Tue, Nov 25, 2014 at 10:30 AM, Flavio Pompermaier <[hidden email]> wrote:

The problem is that RuntimeExceptions are not catched by user code so (IMO) Flink should log somewhere all the stacktrace of those exceptions..

At the moment I see only that during a step there was a NullPointerException but I don't know nothing more (no class or line or stacktrace) :(

On Tue, Nov 25, 2014 at 10:24 AM, Stephan Ewen <[hidden email]> wrote:

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Stephan Ewen]

Can you send us the log of the job manager? There should be an exception stack trace in there... Also the log of the TaskManager on which the task failed should have the exception including trace...

On Tue, Nov 25, 2014 at 11:30 AM, Stefano Bortoli <[hidden email]> wrote:

The problem it is that it is only noted the chain of functions, not the exact function. Furthermore, the same code run several time on a smaller dataset (1M records), but fails on this one (9.2M records). Curiously, the CPUs of the cluster were saturated right before the crush (see picture attached).

saluti,
Stefano

2014-11-25 11:15 GMT+01:00 Flavio Pompermaier <[hidden email]>:

No..they report just a null pointer in a taskmanager during a certain function

On Tue, Nov 25, 2014 at 10:59 AM, Stephan Ewen <[hidden email]> wrote:

The TaskManager log or the JobManager log or the JobManager WebFrontend show nothing?

On Tue, Nov 25, 2014 at 10:30 AM, Flavio Pompermaier <[hidden email]> wrote:

The problem is that RuntimeExceptions are not catched by user code so (IMO) Flink should log somewhere all the stacktrace of those exceptions..

At the moment I see only that during a step there was a NullPointerException but I don't know nothing more (no class or line or stacktrace) :(

On Tue, Nov 25, 2014 at 10:24 AM, Stephan Ewen <[hidden email]> wrote:

Your functions can alway print to sysout or syserr (for example on exceptions). It will be redirected to the *.out files in the Flink log directory.

The TaskManagers themselves should log the exceptions that cause tasks to fail.

The JobManager should also display exceptions that cause failure in the web frontend.

This is not specific to NullPointerExceptions, however.

Does this help?

Am 25.11.2014 09:24 schrieb "Flavio Pompermaier" <[hidden email]>:

Hy to all,

is there a way to print/log all the stacktrace of NullPointerExceptions during the execution of a function in Flink..?

Best,

Flavio

[Stefano Bortoli]

Both, job and task manager do not present any stack trace:

JOBMANAGER
07:13:15,231 INFO  org.apache.flink.runtime.jobmanager.EventCollector            - 11/25/2014 07:13:15: CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map (org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24) switched to FAILED
java.lang.NullPointerException

TASKMANAGER (on the machine that failed)
07:13:15,124 ERROR org.apache.flink.runtime.operators.RegularPactTask            - Error in task code:  CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map
(org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24)
java.lang.NullPointerException

do you want me to attach the whole log files?

saluti,
Stefano

[Stephan Ewen]

Hi!

There is no stack trace beneath that line in the log?

I'll check it out if you send me the log directly ([hidden email]). The mailing list does not support attachments, unfortunately...

Stephan

On Tue, Nov 25, 2014 at 12:01 PM, Stefano Bortoli <[hidden email]> wrote:

Both, job and task manager do not present any stack trace:

JOBMANAGER
07:13:15,231 INFO  org.apache.flink.runtime.jobmanager.EventCollector            - 11/25/2014 07:13:15: CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map (org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24) switched to FAILED
java.lang.NullPointerException

TASKMANAGER (on the machine that failed)
07:13:15,124 ERROR org.apache.flink.runtime.operators.RegularPactTask            - Error in task code:  CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map
(org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24)
java.lang.NullPointerException

do you want me to attach the whole log files?

saluti,
Stefano

[Stefano Bortoli]

nope.

will do. Thanks! :-)

saluti,
Stefano

2014-11-25 12:04 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

There is no stack trace beneath that line in the log?

I'll check it out if you send me the log directly ([hidden email]). The mailing list does not support attachments, unfortunately...

Stephan

On Tue, Nov 25, 2014 at 12:01 PM, Stefano Bortoli <[hidden email]> wrote:

Both, job and task manager do not present any stack trace:

JOBMANAGER
07:13:15,231 INFO  org.apache.flink.runtime.jobmanager.EventCollector            - 11/25/2014 07:13:15: CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map (org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24) switched to FAILED
java.lang.NullPointerException

TASKMANAGER (on the machine that failed)
07:13:15,124 ERROR org.apache.flink.runtime.operators.RegularPactTask            - Error in task code:  CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map
(org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24)
java.lang.NullPointerException

do you want me to attach the whole log files?

saluti,
Stefano

[Stephan Ewen]

Hi!

I can see one NullPointerException in the LOG with the following stack trace 

java.lang.NullPointerException

at java.lang.String.<init>(String.java:556)

at org.okkam.flink.hbase.MyTableInputFormat.mapResultToRecord(MyTableInputFormat.java:90)

at org.okkam.flink.hbase.MyTableInputFormat.getRecord(MyTableInputFormat.java:32)

at org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch.filter(SingleMatchFilterFunctionWithFlagMatch.java:135)

at org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch.filter(SingleMatchFilterFunctionWithFlagMatch.java:50)

at org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)

at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)

at org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction.reduce(DistinctOperator.java:151)

at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:117)

at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)

at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)

at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:246)

at java.lang.Thread.run(Thread.java:745)

There is another one where the stack trace is missing. I am not sure why. Found this here while I searched: http://stackoverflow.com/questions/2411487/nullpointerexception-in-java-with-no-stacktrace

Stephan

On Tue, Nov 25, 2014 at 12:31 PM, Stefano Bortoli <[hidden email]> wrote:

nope.

will do. Thanks! :-)

saluti,
Stefano

2014-11-25 12:04 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

There is no stack trace beneath that line in the log?

I'll check it out if you send me the log directly ([hidden email]). The mailing list does not support attachments, unfortunately...

Stephan

On Tue, Nov 25, 2014 at 12:01 PM, Stefano Bortoli <[hidden email]> wrote:

Both, job and task manager do not present any stack trace:

JOBMANAGER
07:13:15,231 INFO  org.apache.flink.runtime.jobmanager.EventCollector            - 11/25/2014 07:13:15: CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map (org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24) switched to FAILED
java.lang.NullPointerException

TASKMANAGER (on the machine that failed)
07:13:15,124 ERROR org.apache.flink.runtime.operators.RegularPactTask            - Error in task code:  CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map
(org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24)
java.lang.NullPointerException

do you want me to attach the whole log files?

saluti,
Stefano

[Stefano Bortoli]

Sure you found one, but that one happened the day before at noon (11/24/2014 12:28:47). :-) That one was due to actually missing entry in the database. I added a filter to avoid this type of errors.

However, I will checkout the stackoverflow entry. :-)

thanks a lot for the effort anyway.

saluti,
Stefano

2014-11-25 13:44 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

I can see one NullPointerException in the LOG with the following stack trace 

java.lang.NullPointerException

at java.lang.String.<init>(String.java:556)

at org.okkam.flink.hbase.MyTableInputFormat.mapResultToRecord(MyTableInputFormat.java:90)

at org.okkam.flink.hbase.MyTableInputFormat.getRecord(MyTableInputFormat.java:32)

at org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch.filter(SingleMatchFilterFunctionWithFlagMatch.java:135)

at org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch.filter(SingleMatchFilterFunctionWithFlagMatch.java:50)

at org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)

at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)

at org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction.reduce(DistinctOperator.java:151)

at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:117)

at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)

at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)

at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:246)

at java.lang.Thread.run(Thread.java:745)

There is another one where the stack trace is missing. I am not sure why. Found this here while I searched: http://stackoverflow.com/questions/2411487/nullpointerexception-in-java-with-no-stacktrace

Stephan

On Tue, Nov 25, 2014 at 12:31 PM, Stefano Bortoli <[hidden email]> wrote:

nope.

will do. Thanks! :-)

saluti,
Stefano

2014-11-25 12:04 GMT+01:00 Stephan Ewen <[hidden email]>:

Hi!

There is no stack trace beneath that line in the log?

I'll check it out if you send me the log directly ([hidden email]). The mailing list does not support attachments, unfortunately...

Stephan

On Tue, Nov 25, 2014 at 12:01 PM, Stefano Bortoli <[hidden email]> wrote:

Both, job and task manager do not present any stack trace:

JOBMANAGER
07:13:15,231 INFO  org.apache.flink.runtime.jobmanager.EventCollector            - 11/25/2014 07:13:15: CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map (org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24) switched to FAILED
java.lang.NullPointerException

TASKMANAGER (on the machine that failed)
07:13:15,124 ERROR org.apache.flink.runtime.operators.RegularPactTask            - Error in task code:  CHAIN Reduce (org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) -> Filter (org.okkam.flink.maintenance.deduplication.match.SingleMatchFilterFunctionWithFlagMatch) -> Map
(org.okkam.flink.maintenance.deduplication.match.MapToTuple3MapFunction) -> Combine(org.apache.flink.api.java.operators.DistinctOperator$DistinctFunction) (18/24)
java.lang.NullPointerException

do you want me to attach the whole log files?

saluti,
Stefano