How to consume kafka from the last offset?

[Jim Chen]

Hi, All

  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer params, i set the group.id and auto.offset.reset. In the Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();

  Then, i restart the application, found the offset is not from the last position. Any one know where is wrong? HELP!

[Dominik Wosiński]

Hi Jim,

Well, auto.offset.reset is only used when there is no offset saved for this group.id in Kafka. So, if You want to read the data from the latest record (and by latest I mean the newest here) You should assign the group.id that was not previously used and then FlinkKafkaConsumer should automatically fetch the last offset and start reading from that place.

Best Regards,

Dom.

śr., 25 mar 2020 o 11:19 Jim Chen <[hidden email]> napisał(a):

Hi, All

  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer params, i set the group.id and auto.offset.reset. In the Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();

  Then, i restart the application, found the offset is not from the last position. Any one know where is wrong? HELP!

[Jim Chen]

Thanks! 

I made a mistake. I forget to set the auto.offset.reset=false. It's my fault.

Dominik Wosiński <[hidden email]> 于2020年3月25日周三 下午6:49写道：

Hi Jim,

Well, auto.offset.reset is only used when there is no offset saved for this group.id in Kafka. So, if You want to read the data from the latest record (and by latest I mean the newest here) You should assign the group.id that was not previously used and then FlinkKafkaConsumer should automatically fetch the last offset and start reading from that place.

Best Regards,

Dom.

śr., 25 mar 2020 o 11:19 Jim Chen <[hidden email]> napisał(a):

Hi, All

  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer params, i set the group.id and auto.offset.reset. In the Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();

  Then, i restart the application, found the offset is not from the last position. Any one know where is wrong? HELP!

[Dominik Wosiński]

Hey,

Are You completely sure you mean auto.offset.reset ??  False is not valid setting for that AFAIK.

Best,

Dom.

czw., 26 mar 2020 o 08:38 Jim Chen <[hidden email]> napisał(a):

Thanks! 

I made a mistake. I forget to set the auto.offset.reset=false. It's my fault.

Dominik Wosiński <[hidden email]> 于2020年3月25日周三 下午6:49写道：

Hi Jim,

Well, auto.offset.reset is only used when there is no offset saved for this group.id in Kafka. So, if You want to read the data from the latest record (and by latest I mean the newest here) You should assign the group.id that was not previously used and then FlinkKafkaConsumer should automatically fetch the last offset and start reading from that place.

Best Regards,

Dom.

śr., 25 mar 2020 o 11:19 Jim Chen <[hidden email]> napisał(a):

Hi, All

  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer params, i set the group.id and auto.offset.reset. In the Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();

  Then, i restart the application, found the offset is not from the last position. Any one know where is wrong? HELP!

[Jim Chen]

Hi,

I am so sorry. It's not auto.offset.reset. Correctly, it is enable.auto.commit=false

Best Wishs!

Dominik Wosiński <[hidden email]> 于2020年3月26日周四 下午4:20写道：

Hey,

Are You completely sure you mean auto.offset.reset ??  False is not valid setting for that AFAIK.

Best,

Dom.

czw., 26 mar 2020 o 08:38 Jim Chen <[hidden email]> napisał(a):

Thanks! 

I made a mistake. I forget to set the auto.offset.reset=false. It's my fault.

Dominik Wosiński <[hidden email]> 于2020年3月25日周三 下午6:49写道：

Hi Jim,

Well, auto.offset.reset is only used when there is no offset saved for this group.id in Kafka. So, if You want to read the data from the latest record (and by latest I mean the newest here) You should assign the group.id that was not previously used and then FlinkKafkaConsumer should automatically fetch the last offset and start reading from that place.

Best Regards,

Dom.

śr., 25 mar 2020 o 11:19 Jim Chen <[hidden email]> napisał(a):

Hi, All

  I use flink-connector-kafka-0.11 consume the Kafka0.11. In KafkaConsumer params, i set the group.id and auto.offset.reset. In the Flink1.10, set the kafkaConsumer.setStartFromGroupOffsets();

  Then, i restart the application, found the offset is not from the last position. Any one know where is wrong? HELP!