open checkpoint, send message to kafka OutOfOrderSequenceException

[SmileSmile]

Dear all：
          flink version is 1.12.4，kafka version is 1.1.1。topology is very simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a few hours 。 the error message is 


Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: The broker received an out of order sequence number.
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99) 


product's properties is simle ,  using defalut AT_LEAST_ONCE semantic

        properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");
        properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");
        properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "2000");
        properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, "524288");
        properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "100");
        properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "60000");

My questions is how to deal with ？

 

[Yun Gao]

Hi,

Have you checked if the error during normal execution, or right after failover?

Best,

Yun

------------------------------------------------------------------

From:SmileSmile <[hidden email]>

Send Time:2021 Jun. 4 (Fri.) 11:07

To:user <[hidden email]>

Subject:open checkpoint, send message to kafka OutOfOrderSequenceException

Dear all：
          flink version is 1.12.4，kafka version is 1.1.1。topology is very simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a few hours 。 the error message is 


Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: The broker received an out of order sequence number.
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99) 


product's properties is simle ,  using defalut AT_LEAST_ONCE semantic

        properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");
        properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");
        properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "2000");
        properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, "524288");
        properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "100");
        properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "60000");

My questions is how to deal with ？

 

[SmileSmile]

Hi,

 after failover still OutOfOrderSequenceException. when I close checkpoint, kafka broker still return OutOfOrderSequenceException to me .  

At 2021-06-04 17:52:22, "Yun Gao" <[hidden email]> wrote:

Hi,

Have you checked if the error during normal execution, or right after failover?

Best,

Yun

------------------------------------------------------------------

From:SmileSmile <[hidden email]>

Send Time:2021 Jun. 4 (Fri.) 11:07

To:user <[hidden email]>

Subject:open checkpoint, send message to kafka OutOfOrderSequenceException

Dear all：
          flink version is 1.12.4，kafka version is 1.1.1。topology is very simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a few hours 。 the error message is 


Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: The broker received an out of order sequence number.
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850) 
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99) 


product's properties is simle ,  using defalut AT_LEAST_ONCE semantic

        properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");
        properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");
        properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "2000");
        properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, "524288");
        properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "100");
        properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "60000");

My questions is how to deal with ？

 

 

[Yun Gao]

Hi,

I tried google and found one possible related issue in [1]. Could you try to remove the config

properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");

and try again ?

Best,

Yun

[1] https://stackoverflow.com/questions/55192852/transactional-producer-vs-just-idempotent-producer-java-exception-outoforderseq

------------------Original Mail ------------------

Sender:SmileSmile <[hidden email]>

Send Date:Fri Jun 4 23:24:06 2021

Recipients:Yun Gao <[hidden email]>

CC:user <[hidden email]>

Subject:Re:Re: open checkpoint, send message to kafka OutOfOrderSequenceException

Hi,

 after failover still OutOfOrderSequenceException. when I close checkpoint, kafka broker still return OutOfOrderSequenceException to me .  

At 2021-06-04 17:52:22, "Yun Gao" <[hidden email]> wrote:

Hi,

Have you checked if the error during normal execution, or right after failover?

Best,

Yun

------------------------------------------------------------------

From:SmileSmile <[hidden email]>

Send Time:2021 Jun. 4 (Fri.) 11:07

To:user <[hidden email]>

Subject:open checkpoint, send message to kafka OutOfOrderSequenceException

Dear all：          flink version is 1.12.4，kafka version is 1.1.1。topology is very simple ，source-->flatmap--->sink ，enable checkpoint，job will fail after a few hours 。 the error message is Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: The broker received an out of order sequence number.at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1392) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:850) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99) product's properties is simle ,  using defalut AT_LEAST_ONCE semantic

        properties.setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");        properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");        properties.setProperty(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");        properties.setProperty(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "2000");        properties.setProperty(ProducerConfig.BATCH_SIZE_CONFIG, "524288");        properties.setProperty(ProducerConfig.LINGER_MS_CONFIG, "100");        properties.setProperty(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "60000");My questions is how to deal with ？