Measure Operator Runtime in Local Execution Environment

[Stefan Bunk]

Hey Flinkers!

After updating to 0.7-SNAPSHOT I noticed that the runtime of my program when running in the local environment went up by 25%.

That might be due to the refactoring I had to do.

Is there some way to get a report of the runtime of each operator, so that I can narrow down the issue?

Greetings

Stefan

[rmetzger0]

Hi,

were the reference jobs implemented using the old Scala API? Maybe it is faster than the new Scala API?

The webinterface of Flink shows (in the history view) a breakdown of the operator runtimes. You can also set the log level to "INFO" and calculate the operator runtimes yourself.

Remember that operators often run in parallel, which makes it harder to understand the performance behavior.

Please share with us what you found out in your analysis. 

Robert

On Fri, Oct 17, 2014 at 8:29 PM, Stefan Bunk <[hidden email]> wrote:

Hey Flinkers!

After updating to 0.7-SNAPSHOT I noticed that the runtime of my program when running in the local environment went up by 25%.

That might be due to the refactoring I had to do.

Is there some way to get a report of the runtime of each operator, so that I can narrow down the issue?

Greetings

Stefan

[Stephan Ewen]

Hi!

This may be due to many aspects: different startup behavior of the local environment. Different deployment behavior, different shutdown behavior.

Do you know where the time is spent?

Stephan

[Stefan Bunk]

Hi,

yes, I used the old Scala API, and now ported to the new one.

One thing I noticed is the different memory behaviour. 0.7 seems to allocate more memory (but when I look at the documentation for 0.6 and 0.7 the value of taskmanager.memory.fraction did not change), and the garbage collection (therefore?) runs more often.

Please find attached two profiling screenshots of the same flink program, in 0.6 and 0.7.

Is there some way to set the taskmanager.memory.fraction for the local execution?

Stefan

On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

This may be due to many aspects: different startup behavior of the local environment. Different deployment behavior, different shutdown behavior.

Do you know where the time is spent?

Stephan

[Stephan Ewen]

Hey Stefan!

The Apache mailing lists do not take attachments. Can you upload them somewhere (e.g. public dropbox folder or so?)

Stephan

On Fri, Oct 24, 2014 at 3:18 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

yes, I used the old Scala API, and now ported to the new one.

One thing I noticed is the different memory behaviour. 0.7 seems to allocate more memory (but when I look at the documentation for 0.6 and 0.7 the value of taskmanager.memory.fraction did not change), and the garbage collection (therefore?) runs more often.

Please find attached two profiling screenshots of the same flink program, in 0.6 and 0.7.

Is there some way to set the taskmanager.memory.fraction for the local execution?

Stefan

On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

This may be due to many aspects: different startup behavior of the local environment. Different deployment behavior, different shutdown behavior.

Do you know where the time is spent?

Stephan

[Stefan Bunk]

See here: https://www.dropbox.com/sh/r041egk3dt9eev3/AADlkAyJBlPXSANY4rwOjCBBa?dl=0

On Fri, Oct 24, 2014 at 4:30 PM, Stephan Ewen <[hidden email]> wrote:

Hey Stefan!

The Apache mailing lists do not take attachments. Can you upload them somewhere (e.g. public dropbox folder or so?)

Stephan

On Fri, Oct 24, 2014 at 3:18 PM, Stefan Bunk <[hidden email]> wrote:

Hi,

yes, I used the old Scala API, and now ported to the new one.

One thing I noticed is the different memory behaviour. 0.7 seems to allocate more memory (but when I look at the documentation for 0.6 and 0.7 the value of taskmanager.memory.fraction did not change), and the garbage collection (therefore?) runs more often.

Please find attached two profiling screenshots of the same flink program, in 0.6 and 0.7.

Is there some way to set the taskmanager.memory.fraction for the local execution?

Stefan

On Sat, Oct 18, 2014 at 8:52 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

This may be due to many aspects: different startup behavior of the local environment. Different deployment behavior, different shutdown behavior.

Do you know where the time is spent?

Stephan