Restore from save point but need to read from different Kafka topics

[ruben.casado.tejedor]

Hi

 

Let me introduce our scenario:

 

1. We have a Flink job reading from a Kafka topic, using the Flink Kafka. Name of Kafka topic is an input variable in properties file
2. A savepoint is created for that job, so the Kafka offsets for the input topic is stored in that savepoint
3. The job is cancelled
4. The kafka topic from which the job reads is modified in the properties file
5. The flink job is executed from the savepoint

 

What happens in this scenario? Does the job read from the beginning of the new Kafka topic? Or does the job fail?

What we would need is to read from earliest in the new topic. Is it possible?

 

Thanks in advance!

 

------------------------------------------

Rubén Casado Tejedor, PhD

Big Data Lead

> accenture technology

' + 34 629 009 429

* [hidden email]

---

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
______________________________________________________________________________________

www.accenture.com

[rmetzger0]

Hey,

as far as I remember, the Kafka consumer stores the topic name + partition id + offset in state. If you modify the kafka topic on restore, the Kafka consumer would continue where it left off.

If the topics don't exist anymore, it would fail.

If you want Kafka to start from a different topic, you would need to pass the Kafka consumer a different uuid, so that the state of that operator gets discarded.

You can configure Kafka's StartupMode "Group offset, earliest, latest, ..." I guess you would want to use "earliest".

Best,

Robert

On Thu, Apr 23, 2020 at 6:56 PM Casado Tejedor, Rubén <[hidden email]> wrote:

Hi

 

Let me introduce our scenario:

 

1. We have a Flink job reading from a Kafka topic, using the Flink Kafka. Name of Kafka topic is an input variable in properties file
2. A savepoint is created for that job, so the Kafka offsets for the input topic is stored in that savepoint
3. The job is cancelled
4. The kafka topic from which the job reads is modified in the properties file
5. The flink job is executed from the savepoint

 

What happens in this scenario? Does the job read from the beginning of the new Kafka topic? Or does the job fail?

What we would need is to read from earliest in the new topic. Is it possible?

 

Thanks in advance!

 

------------------------------------------

Rubén Casado Tejedor, PhD

Big Data Lead

> accenture technology

' + 34 629 009 429

* [hidden email]

---

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
______________________________________________________________________________________

www.accenture.com