Debugging, logging and measuring operator subtask performance

[Dominik Safaric]

Hi,

As I am experiencing certain performance degradations in a streaming job, I want to determine the root cause of it by measuring subtask performance in terms of resource utilisation - e.g. CPU utilisation of the thread.

Is this somehow possible? Does Flink log scheduled and executed threads? What approach would you recommend?

Thanks in advance,
Dominik

[rmetzger0]

Hi Dominik,

You could measure the throughput at each task in your job to see if one operator is causing the slowdown (for example using Flink's metrics system)

Maybe the backpressure view already helps finding the task that causes the issue.

Did you check if there are enough resources available for the processing you intend to do. How saturated are the CPUs, disks and the network?

Regards,

Robert

On Wed, Jan 25, 2017 at 7:51 PM, Dominik Safaric <[hidden email]> wrote:

Hi,

As I am experiencing certain performance degradations in a streaming job, I want to determine the root cause of it by measuring subtask performance in terms of resource utilisation - e.g. CPU utilisation of the thread.

Is this somehow possible? Does Flink log scheduled and executed threads? What approach would you recommend?

Thanks in advance,
Dominik