workaround for kafka idle partition and watermark

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

workaround for kafka idle partition and watermark

aldu29
Hello,

I've faced to this issue in production: https://issues.apache.org/jira/browse/FLINK-5479
One topic contains idle partition and all pipeline is quite fucked up. I've used aggregation based on these watermarks and the trigger is never launched.
is it possible to define a workaround in waiting the 1.8.0 release that will fix this issue ?

Thanks in advance

Here after the code:
kafkaStream.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<MyEvent>() {
    @Override
    public long extractAscendingTimestamp(MyEvent element) {
        LOG.debug("Timestamp event in Kafka: {}", element.getTimestamp());
        return Long.parseLong(element.getTimestamp());
    }
});