Login  Register

Accumulator with Elasticsearch Sink

classic Classic list List threaded Threaded
3 messages Options Options
Embed post
Permalink
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Accumulator with Elasticsearch Sink

Hung
Hi Flink users,

Did someone use accumulator with Elasticsearch Sink? So we can better
compare the last timestamps in the sink and the last timestamps in
Elasticsearch, in order to see how long does it take from the Elasticsearch
sink to Elasticsearch.

Best,

Sendoh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Accumulator with Elasticsearch Sink

Tzu-Li (Gordon) Tai
Hi Sendoh,

That sounds like a reasonable metric to add directly to the Elasticsearch connector.
Could you perhaps write a comment on that in https://issues.apache.org/jira/browse/FLINK-7697?

Cheers,
Gordon

On 19 October 2017 at 8:57:23 PM, Sendoh ([hidden email]) wrote:

Hi Flink users,

Did someone use accumulator with Elasticsearch Sink? So we can better
compare the last timestamps in the sink and the last timestamps in
Elasticsearch, in order to see how long does it take from the Elasticsearch
sink to Elasticsearch.

Best,

Sendoh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Accumulator with Elasticsearch Sink

Hung
This post was updated on Oct 23, 2017; 2:44pm.
Hi Gordon,

I commented the ticket. What would you think I start to implement it as
there is no progress recently?

I see it's only about adding metrics in the following class? and anyway I
probably would have to build it soon.

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java

The other thing I find is my usecase needs accumulator than metric, as the global status is more useful for recognizing the latency of the job, the same as the following discussion:
http://mail-archives.apache.org/mod_mbox/flink-dev/201609.mbox/%3C57E4FCA4.6080103@apache.org%3E

Cheers,

Sendoh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/