(DEPRECATED) Apache Flink User Mailing List archive.

JVM core dump running Flink streaming

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Jakob Ericsson
Reply | Threaded
Open this post in threaded view
♦
♦
|

JVM core dump running Flink streaming

Jakob Ericsson
Hi,

I'm testing Flink streaming but seems to have some problems with jvm core dumps.
I haven't really looked at the heap dump yet.
It seems to be related to the G1 GC.

If I want to go back to CMS-GC, do you have any preferred settings?




#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa281df900d, pid=36541, tid=140335655700224
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5b900d]  G1ParScanThreadState::copy_to_survivor_space(oopDesc*)+0x3d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa26c00f800):  GCTaskThread [stack: 0x00007fa270d84000,0x00007fa270e85000] [id=36559]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

Registers:
RAX=0x00007fa2827b03b0, RBX=0x00007fa2827b03a0, RCX=0x0000000000000003, RDX=0x0000000000000001
RSP=0x00007fa270e83490, RBP=0x00007fa270e83520, RSI=0x00000000dc6e5530, RDI=0x0000000000000000
R8 =0x00000000dc6e5530, R9 =0x00007fa282929ba7, R10=0x00000000c49742a8, R11=0x00007fa2822b2e20
R12=0x000000008c579484, R13=0x00000000dc6e5530, R14=0x00007fa270e83780, R15=0x00007fa270e83aa0
RIP=0x00007fa281df900d, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa270e83490)
0x00007fa270e83490:   00007fa270e834f0 00000000c49742c0
0x00007fa270e834a0:   00007fa270e83780 000000008210a22f
0x00007fa270e834b0:   000000000000000b 00007fa26c333110
0x00007fa270e834c0:   0000000700000001 0000000000000001
0x00007fa270e834d0:   000000000257e760 000000000257e760
0x00007fa270e834e0:   00007fa270e83790 00007fa23c37da70
0x00007fa270e834f0:   00007fa270e83520 00007fa2827b03a0
0x00007fa270e83500:   000000008c579484 00007fa26c0f7e20
0x00007fa270e83510:   00007fa270e83780 00007fa270e83aa0
...
...
...
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: JVM core dump running Flink streaming

Till Rohrmann

Hi Jakob,

you can control the used garbage collector by specifying the env.java.opts configuration parameter in your flink-conf.yaml. Here you can basically specify all the JVM options which will be given to the JVMs. Thus, in your case you could try the following settings: env.java.opts: -XX:+UseConcMarkSweepGC.

Cheers,
Till

​

On Mon, Sep 28, 2015 at 10:15 AM, Jakob Ericsson <[hidden email]> wrote:
Hi,

I'm testing Flink streaming but seems to have some problems with jvm core dumps.
I haven't really looked at the heap dump yet.
It seems to be related to the G1 GC.

If I want to go back to CMS-GC, do you have any preferred settings?




#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa281df900d, pid=36541, tid=140335655700224
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5b900d]  G1ParScanThreadState::copy_to_survivor_space(oopDesc*)+0x3d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa26c00f800):  GCTaskThread [stack: 0x00007fa270d84000,0x00007fa270e85000] [id=36559]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

Registers:
RAX=0x00007fa2827b03b0, RBX=0x00007fa2827b03a0, RCX=0x0000000000000003, RDX=0x0000000000000001
RSP=0x00007fa270e83490, RBP=0x00007fa270e83520, RSI=0x00000000dc6e5530, RDI=0x0000000000000000
R8 =0x00000000dc6e5530, R9 =0x00007fa282929ba7, R10=0x00000000c49742a8, R11=0x00007fa2822b2e20
R12=0x000000008c579484, R13=0x00000000dc6e5530, R14=0x00007fa270e83780, R15=0x00007fa270e83aa0
RIP=0x00007fa281df900d, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa270e83490)
0x00007fa270e83490:   00007fa270e834f0 00000000c49742c0
0x00007fa270e834a0:   00007fa270e83780 000000008210a22f
0x00007fa270e834b0:   000000000000000b 00007fa26c333110
0x00007fa270e834c0:   0000000700000001 0000000000000001
0x00007fa270e834d0:   000000000257e760 000000000257e760
0x00007fa270e834e0:   00007fa270e83790 00007fa23c37da70
0x00007fa270e834f0:   00007fa270e83520 00007fa2827b03a0
0x00007fa270e83500:   000000008c579484 00007fa26c0f7e20
0x00007fa270e83510:   00007fa270e83780 00007fa270e83aa0
...
...
...
Jakob Ericsson
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: JVM core dump running Flink streaming

Jakob Ericsson
Yes. I just wanted to know if you have any preferred settings for CMS or should I use fairly standard configs?

On Mon, Sep 28, 2015 at 11:19 AM, Till Rohrmann <[hidden email]> wrote:

Hi Jakob,

you can control the used garbage collector by specifying the env.java.opts configuration parameter in your flink-conf.yaml. Here you can basically specify all the JVM options which will be given to the JVMs. Thus, in your case you could try the following settings: env.java.opts: -XX:+UseConcMarkSweepGC.

Cheers,
Till

​

On Mon, Sep 28, 2015 at 10:15 AM, Jakob Ericsson <[hidden email]> wrote:
Hi,

I'm testing Flink streaming but seems to have some problems with jvm core dumps.
I haven't really looked at the heap dump yet.
It seems to be related to the G1 GC.

If I want to go back to CMS-GC, do you have any preferred settings?




#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa281df900d, pid=36541, tid=140335655700224
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5b900d]  G1ParScanThreadState::copy_to_survivor_space(oopDesc*)+0x3d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa26c00f800):  GCTaskThread [stack: 0x00007fa270d84000,0x00007fa270e85000] [id=36559]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

Registers:
RAX=0x00007fa2827b03b0, RBX=0x00007fa2827b03a0, RCX=0x0000000000000003, RDX=0x0000000000000001
RSP=0x00007fa270e83490, RBP=0x00007fa270e83520, RSI=0x00000000dc6e5530, RDI=0x0000000000000000
R8 =0x00000000dc6e5530, R9 =0x00007fa282929ba7, R10=0x00000000c49742a8, R11=0x00007fa2822b2e20
R12=0x000000008c579484, R13=0x00000000dc6e5530, R14=0x00007fa270e83780, R15=0x00007fa270e83aa0
RIP=0x00007fa281df900d, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa270e83490)
0x00007fa270e83490:   00007fa270e834f0 00000000c49742c0
0x00007fa270e834a0:   00007fa270e83780 000000008210a22f
0x00007fa270e834b0:   000000000000000b 00007fa26c333110
0x00007fa270e834c0:   0000000700000001 0000000000000001
0x00007fa270e834d0:   000000000257e760 000000000257e760
0x00007fa270e834e0:   00007fa270e83790 00007fa23c37da70
0x00007fa270e834f0:   00007fa270e83520 00007fa2827b03a0
0x00007fa270e83500:   000000008c579484 00007fa26c0f7e20
0x00007fa270e83510:   00007fa270e83780 00007fa270e83aa0
...
...
...
Till Rohrmann-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: JVM core dump running Flink streaming

Till Rohrmann-2

When the CMS garbage collector is used we additionally set -XX:+CMSClassUnloadingEnabled. Everything else are the default values.

​

On Mon, Sep 28, 2015 at 11:22 AM, Jakob Ericsson <[hidden email]> wrote:
Yes. I just wanted to know if you have any preferred settings for CMS or should I use fairly standard configs?

On Mon, Sep 28, 2015 at 11:19 AM, Till Rohrmann <[hidden email]> wrote:

Hi Jakob,

you can control the used garbage collector by specifying the env.java.opts configuration parameter in your flink-conf.yaml. Here you can basically specify all the JVM options which will be given to the JVMs. Thus, in your case you could try the following settings: env.java.opts: -XX:+UseConcMarkSweepGC.

Cheers,
Till

​

On Mon, Sep 28, 2015 at 10:15 AM, Jakob Ericsson <[hidden email]> wrote:
Hi,

I'm testing Flink streaming but seems to have some problems with jvm core dumps.
I haven't really looked at the heap dump yet.
It seems to be related to the G1 GC.

If I want to go back to CMS-GC, do you have any preferred settings?




#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa281df900d, pid=36541, tid=140335655700224
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5b900d]  G1ParScanThreadState::copy_to_survivor_space(oopDesc*)+0x3d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa26c00f800):  GCTaskThread [stack: 0x00007fa270d84000,0x00007fa270e85000] [id=36559]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000008

Registers:
RAX=0x00007fa2827b03b0, RBX=0x00007fa2827b03a0, RCX=0x0000000000000003, RDX=0x0000000000000001
RSP=0x00007fa270e83490, RBP=0x00007fa270e83520, RSI=0x00000000dc6e5530, RDI=0x0000000000000000
R8 =0x00000000dc6e5530, R9 =0x00007fa282929ba7, R10=0x00000000c49742a8, R11=0x00007fa2822b2e20
R12=0x000000008c579484, R13=0x00000000dc6e5530, R14=0x00007fa270e83780, R15=0x00007fa270e83aa0
RIP=0x00007fa281df900d, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa270e83490)
0x00007fa270e83490:   00007fa270e834f0 00000000c49742c0
0x00007fa270e834a0:   00007fa270e83780 000000008210a22f
0x00007fa270e834b0:   000000000000000b 00007fa26c333110
0x00007fa270e834c0:   0000000700000001 0000000000000001
0x00007fa270e834d0:   000000000257e760 000000000257e760
0x00007fa270e834e0:   00007fa270e83790 00007fa23c37da70
0x00007fa270e834f0:   00007fa270e83520 00007fa2827b03a0
0x00007fa270e83500:   000000008c579484 00007fa26c0f7e20
0x00007fa270e83510:   00007fa270e83780 00007fa270e83aa0
...
...
...
Free forum by Nabble Edit this page