KafkaSourceBuilder causing invalid negative offset on checkpointing

[Lars Skjærven-2]

Hello, 

I ran into an issue when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. 

The configuration 'client.id.prefix' was supplied but isn't a known config.       
The configuration 'partition.discovery.interval.ms' was supplied but isn't a known config.        

Finally the job crashed with a checkpointing error:

java.lang.Exception: Could not perform checkpoint 10 for operator Source: progress source (4/6)#9.
....
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 10 for operator Source: progress source (4/6)#9. Failure reason: Checkpoint was declined.
...
Caused by: java.lang.IllegalArgumentException: Invalid negative offset

Switching back to using FlinkKafkaConsumer, the warnings on the kafka config disappeared, and the job was able to checkpoint successfully.

I'm wondering if the warnings and the errors are related, and if there is a compatibility issue between the newer KafkaSourceBuilder and Kafka 2.5 ? 

Thanks, 

L

[Till Rohrmann]

Hi Lars,

I think this is a duplicate message. Let's continue the discussion on your original message.

Cheers,

Till

On Wed, Apr 28, 2021 at 8:50 PM Lars Skjærven <[hidden email]> wrote:

Hello, 

I ran into an issue when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. 

The configuration 'client.id.prefix' was supplied but isn't a known config.       
The configuration 'partition.discovery.interval.ms' was supplied but isn't a known config.        

Finally the job crashed with a checkpointing error:

java.lang.Exception: Could not perform checkpoint 10 for operator Source: progress source (4/6)#9.
....
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 10 for operator Source: progress source (4/6)#9. Failure reason: Checkpoint was declined.
...
Caused by: java.lang.IllegalArgumentException: Invalid negative offset

Switching back to using FlinkKafkaConsumer, the warnings on the kafka config disappeared, and the job was able to checkpoint successfully.

I'm wondering if the warnings and the errors are related, and if there is a compatibility issue between the newer KafkaSourceBuilder and Kafka 2.5 ? 

Thanks, 

L