Flink Kafka from Time Offset

[simon peyer]

Hi together

I'm using flink 1.0.1 and a FlinkKafkaConsumer09.

I'm very interested in getting data from a specific Time offset in Kafka.

Is there a property which can do this?

Or is there another way of handling such issues?

I'm also using checkpointing.

If I deploy a new pipeline with the same id's on the nodes, will the checkpoint be atomatically loaded and the state restored?

-Thanks

Simon

[rmetzger0]

Hi Simon,

Timestamp-awareness has been added to Kafka 0.10 only [1]. I'm not sure if the 0.9 connector code of Flink will support Kafka 0.10 immediately.

Another way of handling the issue would be 

- Implementing a custom offset-metadata system (storing the timestamp for some of the offsets (say every minute or so))

- Implement this feature into Flink's Kafka connector: https://issues.apache.org/jira/browse/FLINK-3123

Regarding your second question: What "id"s do you mean?

If you are not using savepoints, the checkpoint state will not be automatically restored.

Regards,

Robert

[1] https://issues.apache.org/jira/browse/KAFKA-2511

On Mon, May 30, 2016 at 1:45 PM, simon peyer <[hidden email]> wrote:

Hi together

I'm using flink 1.0.1 and a FlinkKafkaConsumer09.

I'm very interested in getting data from a specific Time offset in Kafka.

Is there a property which can do this?

Or is there another way of handling such issues?

I'm also using checkpointing.

If I deploy a new pipeline with the same id's on the nodes, will the checkpoint be atomatically loaded and the state restored?

-Thanks

Simon