Re: Flink-Kafka connector - partition offsets for a given timestamp?

Posted by Tzu-Li (Gordon) Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-Kafka-connector-partition-offsets-for-a-given-timestamp-tp17187p17188.html

Hi Connie,

We do have a pull request for the feature, that should almost be ready after rebasing: https://github.com/apache/flink/pull/3915, JIRA: https://issues.apache.org/jira/browse/FLINK-6352.
This means, of course, that the feature isn't part of any release yet. We can try to make sure this happens for Flink 1.5, for which the proposed release date is around February 2018.

Cheers,
Gordon

On Tue, Dec 12, 2017 at 3:53 AM, Yang, Connie <[hidden email]> wrote:

Hi,

 

Does Flink-Kafka connector allow job graph to consume topoics/partitions from a specific timestamp?

 

https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L469 seems to suggest that a job graph can only start from an earliest, latest or a set of offsets.

 

KafkaConsumer API, https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1598, gives us a way to find partition offsets based on a timestamp.

 

Thanks

Connie