Flink-Kafka connector - partition offsets for a given timestamp?

[Yang, Connie]

Hi,

 

Does Flink-Kafka connector allow job graph to consume topoics/partitions from a specific timestamp?

 

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L469 seems to suggest that a job graph can only start from an earliest, latest or a set of offsets.

 

KafkaConsumer API, https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1598, gives us a way to find partition offsets based on a timestamp.

 

Thanks

Connie

[Tzu-Li (Gordon) Tai]

Hi Connie,

We do have a pull request for the feature, that should almost be ready after rebasing: https://github.com/apache/flink/pull/3915, JIRA: https://issues.apache.org/jira/browse/FLINK-6352.
This means, of course, that the feature isn't part of any release yet. We can try to make sure this happens for Flink 1.5, for which the proposed release date is around February 2018.

Cheers,
Gordon

On Tue, Dec 12, 2017 at 3:53 AM, Yang, Connie <[hidden email]> wrote:

Hi,

 

Does Flink-Kafka connector allow job graph to consume topoics/partitions from a specific timestamp?

 

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L469 seems to suggest that a job graph can only start from an earliest, latest or a set of offsets.

 

KafkaConsumer API, https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1598, gives us a way to find partition offsets based on a timestamp.

 

Thanks

Connie

... [show rest of quote]

[Yang, Connie]

Thanks, Gordan!  Will keep an eye on that!

 

Connie

 

From: "Tzu-Li (Gordon) Tai" <[hidden email]>
Date: Monday, December 11, 2017 at 5:29 PM
To: Connie Yang <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Flink-Kafka connector - partition offsets for a given timestamp?

 

Hi Connie,

We do have a pull request for the feature, that should almost be ready after rebasing: https://github.com/apache/flink/pull/3915, JIRA: https://issues.apache.org/jira/browse/FLINK-6352.
This means, of course, that the feature isn't part of any release yet. We can try to make sure this happens for Flink 1.5, for which the proposed release date is around February 2018.

Cheers,
Gordon

 

On Tue, Dec 12, 2017 at 3:53 AM, Yang, Connie <[hidden email]> wrote:

Hi,

 

Does Flink-Kafka connector allow job graph to consume topoics/partitions from a specific timestamp?

 

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L469 seems to suggest that a job graph can only start from an earliest, latest or a set of offsets.

 

KafkaConsumer API, https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1598, gives us a way to find partition offsets based on a timestamp.

 

Thanks

Connie

... [show rest of quote]