streaming file sink OUT metrics not populating

[Vijayendra Yadav]

Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,

Vijay

[Chesnay Schepler]

This is a known issue, and cannot be fixed on the user side.

The underlying problem is that this needs to be implemented separately for each source/sink and we haven't gotten around to doing that yet, but some progress is being made for 1.14 to that end.

On 6/3/2021 6:06 PM, Vijayendra Yadav wrote:

Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,

Vijay

[Arvid Heise-4]

For reference, the respective FLIP shows the ideas [1]. It's on our agenda for 1.14.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics

On Thu, Jun 3, 2021 at 6:41 PM Chesnay Schepler <[hidden email]> wrote:

This is a known issue, and cannot be fixed on the user side.

The underlying problem is that this needs to be implemented separately for each source/sink and we haven't gotten around to doing that yet, but some progress is being made for 1.14 to that end.

On 6/3/2021 6:06 PM, Vijayendra Yadav wrote:

Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,

Vijay