KakfaConsumer

[Vishal Santoshi]

A simple query

Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?  Note that the I do koow that Flink consumes data from Kafka topics and periodically checkpoints using Flink's distributed checkpointing mechanism. In case of failure, Flink will restore the records from checkpoint directory and will start reading data from Kafka offset after that. In this case however I want to restart a radically different pipe but from an offset that is part of the SP and the hope is that I could just depend on the kafka offsets of the consumer group. 

Thanks

[Nagarjun Guraja]

Hi Vishal,

Flink does checkpoint to Kafka(Offset commits) by default which could be disabled. Look here for more information.

On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <[hidden email]> wrote:

A simple query

Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?  Note that the I do koow that Flink consumes data from Kafka topics and periodically checkpoints using Flink's distributed checkpointing mechanism. In case of failure, Flink will restore the records from checkpoint directory and will start reading data from Kafka offset after that. In this case however I want to restart a radically different pipe but from an offset that is part of the SP and the hope is that I could just depend on the kafka offsets of the consumer group. 

Thanks

[Vishal Santoshi]

So it  does that also commit the offsets to kafka on checkpoint/savepoint as well to it's own distributed state ? Just wanted to confirm. 

On Mon, Feb 4, 2019 at 6:56 PM Nagarjun Guraja <[hidden email]> wrote:

Hi Vishal,

Flink does checkpoint to Kafka(Offset commits) by default which could be disabled. Look here for more information.

On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <[hidden email]> wrote:

A simple query

Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?  Note that the I do koow that Flink consumes data from Kafka topics and periodically checkpoints using Flink's distributed checkpointing mechanism. In case of failure, Flink will restore the records from checkpoint directory and will start reading data from Kafka offset after that. In this case however I want to restart a radically different pipe but from an offset that is part of the SP and the hope is that I could just depend on the kafka offsets of the consumer group. 

Thanks

[Vishal Santoshi]

In fact "Checkpointing enabled: if checkpointing is enabled, the Flink Kafka Consumer will commit the offsets stored in the checkpointed states when the checkpoints are completed. This ensures that the committed offsets in Kafka brokers is consistent with the offsets in the checkpointed states."  confirms that. 

On Mon, Feb 4, 2019 at 7:29 PM Vishal Santoshi <[hidden email]> wrote:

So it  does that also commit the offsets to kafka on checkpoint/savepoint as well to it's own distributed state ? Just wanted to confirm. 

On Mon, Feb 4, 2019 at 6:56 PM Nagarjun Guraja <[hidden email]> wrote:

Hi Vishal,

Flink does checkpoint to Kafka(Offset commits) by default which could be disabled. Look here for more information.

On Mon, Feb 4, 2019 at 3:31 PM Vishal Santoshi <[hidden email]> wrote:

A simple query

Does Flink' KafkaConnector flush the current offsets to kafka on a SP ?  Note that the I do koow that Flink consumes data from Kafka topics and periodically checkpoints using Flink's distributed checkpointing mechanism. In case of failure, Flink will restore the records from checkpoint directory and will start reading data from Kafka offset after that. In this case however I want to restart a radically different pipe but from an offset that is part of the SP and the hope is that I could just depend on the kafka offsets of the consumer group. 

Thanks