Flink sent/received bytes related metrics: how are they calculated?

[Andrea Spina]

Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at bytes sent and bytes received metrics: what I am recording is the following:

I am reading from a Kafka topic A records with schema K using a source Sin belonging to a pipeline P1. The topic A is filled with K data using a separated pipeline p2 and a sink Sout.

Basically what I register is that the bytes sent metric of Sin - available through the Flink UI -measure twice or more respect of the bytes received of Sout. For instance, if my sink Sout records 10GB of bytes received inbound, then my source Sin emits between 20-25GB bytes sent.

Is someone able to detail how these two metrics are calculated? 

Thank you,

--

[Chesnay Schepler]

Which version of Flink are you using? There were some issues at some point about double-counting.

On 14/06/2019 09:49, Andrea Spina wrote:

Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at bytes sent and bytes received metrics: what I am recording is the following:

I am reading from a Kafka topic A records with schema K using a source Sin belonging to a pipeline P1. The topic A is filled with K data using a separated pipeline p2 and a sink Sout.

Basically what I register is that the bytes sent metric of Sin - available through the Flink UI -measure twice or more respect of the bytes received of Sout. For instance, if my sink Sout records 10GB of bytes received inbound, then my source Sin emits between 20-25GB bytes sent.

Is someone able to detail how these two metrics are calculated? 

Thank you,

--

[Andrea Spina]

Sorry, I totally missed the version: flink-1.6.4, Streaming API

Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler <[hidden email]> ha scritto:

Which version of Flink are you using? There were some issues at some point about double-counting.

On 14/06/2019 09:49, Andrea Spina wrote:

Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at bytes sent and bytes received metrics: what I am recording is the following:

I am reading from a Kafka topic A records with schema K using a source Sin belonging to a pipeline P1. The topic A is filled with K data using a separated pipeline p2 and a sink Sout.

Basically what I register is that the bytes sent metric of Sin - available through the Flink UI -measure twice or more respect of the bytes received of Sout. For instance, if my sink Sout records 10GB of bytes received inbound, then my source Sin emits between 20-25GB bytes sent.

Is someone able to detail how these two metrics are calculated? 

Thank you,

--

--

[Chesnay Schepler]

How does the P1 pipeline look like? Are there 2 downstream operators reading from Sin (in this case the number of bytes would be measured twice)?

On 14/06/2019 12:09, Andrea Spina wrote:

Sorry, I totally missed the version: flink-1.6.4, Streaming API

Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler <[hidden email]> ha scritto:

Which version of Flink are you using? There were some issues at some point about double-counting.

On 14/06/2019 09:49, Andrea Spina wrote:

Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at bytes sent and bytes received metrics: what I am recording is the following:

I am reading from a Kafka topic A records with schema K using a source Sin belonging to a pipeline P1. The topic A is filled with K data using a separated pipeline p2 and a sink Sout.

Basically what I register is that the bytes sent metric of Sin - available through the Flink UI -measure twice or more respect of the bytes received of Sout. For instance, if my sink Sout records 10GB of bytes received inbound, then my source Sin emits between 20-25GB bytes sent.

Is someone able to detail how these two metrics are calculated? 

Thank you,

--

--

[Andrea Spina]

Hi Chesnay, just one downstream: Sin (Source: Enriched Code) outcome is the right part of the following operator as in the figure; this operator is the exclusive downstream of Sin.
Thanks,

Il giorno ven 14 giu 2019 alle ore 12:23 Chesnay Schepler <[hidden email]> ha scritto:

How does the P1 pipeline look like? Are there 2 downstream operators reading from Sin (in this case the number of bytes would be measured twice)?

On 14/06/2019 12:09, Andrea Spina wrote:

Sorry, I totally missed the version: flink-1.6.4, Streaming API

Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler <[hidden email]> ha scritto:

Which version of Flink are you using? There were some issues at some point about double-counting.

On 14/06/2019 09:49, Andrea Spina wrote:

Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at bytes sent and bytes received metrics: what I am recording is the following:

I am reading from a Kafka topic A records with schema K using a source Sin belonging to a pipeline P1. The topic A is filled with K data using a separated pipeline p2 and a sink Sout.

Basically what I register is that the bytes sent metric of Sin - available through the Flink UI -measure twice or more respect of the bytes received of Sout. For instance, if my sink Sout records 10GB of bytes received inbound, then my source Sin emits between 20-25GB bytes sent.

Is someone able to detail how these two metrics are calculated? 

Thank you,

--

--

--