Logs show `Marking the coordinator 2147483637 dead` in Flink-Kafka conn

[Hung]

Hi Flink community,

In worker's log of Flink I saw the following info appears 30 times or more every 10 minutes approximately.
`org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Marking the coordinator 2147483637 dead. `

Would this indicate Kafka-Flink consumer group is re-balancing too much and sacrifices some performance? Because the Heartbeat Test cannot be finished in the session interval? We use the default session interval in Kafka.

Our Flink is 1.0 and Kafka is 0.9, and the Flink consumer uses the following params:
properties.setProperty("bootstrap.servers", Config.bootstrapServers);
properties.setProperty("group.id", parameter.getRequired("groupId"));
properties.setProperty("auto.offset.reset", Config.autoOffsetReset);

Best,

Sendoh

[rmetzger0]

Hi,

the error is logged is coming from the Kafka code we are referring to. The Kafka user@ list is probably a better place to seek for help on this one.

I've searched for the error message at Google. It seems that the message does not immediately indicate an error. Also note, it is logged at level INFO, not WARN or anything severe.

Are there any other warnings logged? Does the error only happen together with other events?

I don't think that this slows down the performance.

Regards,

Robert

On Mon, Jun 6, 2016 at 2:10 PM, Sendoh <[hidden email]> wrote:

Hi Flink community,

In worker's log of Flink I saw the following info appears 30 times or more
every 10 minutes approximately.
`org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Marking
the coordinator 2147483637 dead. `

Would this indicate Kafka-Flink consumer group is re-balancing too much and
sacrifices some performance? Because the Heartbeat Test cannot be finished
in the session interval? We use the default session interval in Kafka.

Our Flink is 1.0 and Kafka is 0.9, and the Flink consumer uses the following
params:
properties.setProperty("bootstrap.servers", Config.bootstrapServers);
properties.setProperty("group.id", parameter.getRequired("groupId"));
properties.setProperty("auto.offset.reset", Config.autoOffsetReset);

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Logs-show-Marking-the-coordinator-2147483637-dead-in-Flink-Kafka-conn-tp7396.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Hung]

Hi Robert,

Thank you for checking the issue. That INFO is the only information Flink workers say.

I agree your point of view. Looks like it closes the connections to all other topics which is not used(idle) although it's a bit misleading.

Ref: https://github.com/edenhill/librdkafka/issues/437

Best,

Sendoh