how to count kafka sink number

hi 

i have a flink sql, reading record from kafka, then use table function do some transformation, then produce to kafka. 

i have found that in the flink web  record received of  the first subTask is always 0 ,and   the Records send of the last subTask is 0 as well.

i want to count how many records i consumed from the source kafka and how many records i produce to the target kafka,

how can i get these information from flink directly?

best regards

chong

 

[Konstantin Knauf-2]

Hi Chong, 

to my knowledge, neither Flink's built-in metrics nor the metrics of the Kafka producer itself give you this number directly. If your sink is chained (no serialization, no network) to another Flink operator, you could take the numRecordsOut of this operator instead. It will tell you how many records have been received by the sink, not how many were produced to the Kafka cluster, though. 

Best, 

Konstantin

 

On Mon, May 13, 2019 at 7:58 AM [hidden email] <[hidden email]> wrote:

hi 

i have a flink sql, reading record from kafka, then use table function do some transformation, then produce to kafka. 

i have found that in the flink web  record received of  the first subTask is always 0 ,and   the Records send of the last subTask is 0 as well.

i want to count how many records i consumed from the source kafka and how many records i produce to the target kafka,

how can i get these information from flink directly?

best regards

chong

 

--