Flink Kafka Connector behaviour on leader change and broker upgrade

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Kafka Connector behaviour on leader change and broker upgrade

Janardhan Reddy
HI,

Does the flink kafka connector 0.8.2 handle broker's leader change gracefully since simple kafka consumer should be handling leader changes for a partition.

How would the consumer behave when upgrading the brokers from 0.8 to 0.9.


Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Flink Kafka Connector behaviour on leader change and broker upgrade

rmetzger0
Hi,
yes, the Flink Kafka connector for Kafka 0.8 handles broker leader changes without failing. The SimpleConsumer provided by Kafka 0.8 doesn't handle that.

The 0.9 Flink Kafka consumer also supports broker leader changes transparently.
If you keep using the Flink Kafka 0.8 connector with a 0.9 broker, you will need to change the protocol version of the 0.9 broker, so that the old client can talk to it. I would recommend using matching consumer and broker versions.

Regards,
Robert


On Fri, Nov 4, 2016 at 11:19 AM, Janardhan Reddy <[hidden email]> wrote:
HI,

Does the flink kafka connector 0.8.2 handle broker's leader change gracefully since simple kafka consumer should be handling leader changes for a partition.

How would the consumer behave when upgrading the brokers from 0.8 to 0.9.


Thanks