Any metrics to get the shuffled and intermediate data in flink

[Darshan Singh]

Hi

Is there any useful metrics in flink which tells me that a given operator read say 1 GB of data and shuffled(or anything else) and written(in case it was written to temp or anywhere else) say 1 or 2 GB data.

One of my job is failing with disk space and there are many sort, group and join is happening and I would want to know which one is generating most of the temp space.

Thanks

[Michael Latta]

If you look at the web UI for flink it will tell you the bytes received and sent for each stage of a job.  I have not seen any similar metric for persisted state per stage, which would be nice to have as well.

Michael

> On Apr 13, 2018, at 6:37 AM, Darshan Singh <[hidden email]> wrote:
>
> Hi
>
> Is there any useful metrics in flink which tells me that a given operator read say 1 GB of data and shuffled(or anything else) and written(in case it was written to temp or anywhere else) say 1 or 2 GB data.
>
> One of my job is failing with disk space and there are many sort, group and join is happening and I would want to know which one is generating most of the temp space.
>
>
> Thanks

[Darshan Singh]

Thanks, I could see those on UI.

Thanks

On Fri, Apr 13, 2018 at 3:12 PM, TechnoMage <[hidden email]> wrote:

If you look at the web UI for flink it will tell you the bytes received and sent for each stage of a job.  I have not seen any similar metric for persisted state per stage, which would be nice to have as well.

Michael

> On Apr 13, 2018, at 6:37 AM, Darshan Singh <[hidden email]> wrote:
>
> Hi
>
> Is there any useful metrics in flink which tells me that a given operator read say 1 GB of data and shuffled(or anything else) and written(in case it was written to temp or anywhere else) say 1 or 2 GB data.
>
> One of my job is failing with disk space and there are many sort, group and join is happening and I would want to know which one is generating most of the temp space.
>
>
> Thanks