streaming file sink OUT metrics not populating

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

streaming file sink OUT metrics not populating

Vijayendra Yadav
Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,
Vijay

Reply | Threaded
Open this post in threaded view
|

Re: streaming file sink OUT metrics not populating

Chesnay Schepler
This is a known issue, and cannot be fixed on the user side.
The underlying problem is that this needs to be implemented separately for each source/sink and we haven't gotten around to doing that yet, but some progress is being made for 1.14 to that end.

On 6/3/2021 6:06 PM, Vijayendra Yadav wrote:
Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,
Vijay


Reply | Threaded
Open this post in threaded view
|

Re: streaming file sink OUT metrics not populating

Arvid Heise-4
For reference, the respective FLIP shows the ideas [1]. It's on our agenda for 1.14.


On Thu, Jun 3, 2021 at 6:41 PM Chesnay Schepler <[hidden email]> wrote:
This is a known issue, and cannot be fixed on the user side.
The underlying problem is that this needs to be implemented separately for each source/sink and we haven't gotten around to doing that yet, but some progress is being made for 1.14 to that end.

On 6/3/2021 6:06 PM, Vijayendra Yadav wrote:
Hi Team,

I am using streaming file sink and sinking data to s3. Also, using Graphite exporter for metrics.

I can see correct counters for Source, Map, Filter functions. But for SINK, only numRecordsIn is populating, I am looking to get numRecordsOut counts also, but it's always staying 0 although data is sinking to s3 correctly.

Any reason for OUT ( numRecordsOut) metrics to stay zero all the time? Any way to fix it ?

Thanks,
Vijay