Re: understanding task manager logs

Posted by Vishwas Siravara on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/understanding-task-manager-logs-tp29775p29855.html

Thanks, I'll check it out.

On Thu, Sep 5, 2019 at 5:22 AM Fabian Hueske <[hidden email]> wrote:

Hi Vishwas,

This is a log statement from Kafka [1].

Not sure how when AppInfoParser is created (the log message is written by the constructor).

For Kafka versions > 1.0, I'd recommend the universal connector [2].

Not sure how well it works if producers and consumers have different versions.

Maybe Gordon (in CC) has some experience with that.

Best, Fabian

[1] https://github.com/axbaretto/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/AppInfoParser.java#L117

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html#kafka-100-connector

Am Di., 3. Sept. 2019 um 04:04 Uhr schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I am using flink 1.7.2 and my application consumes from a kafka topic and publish to another kafka topic which is in its own kafka environment running a different kafka version,. I am using FlinkKafkaConsumer010 from this dependency 

"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. 

In the task manager log I see these lines: 

2019-09-02 02:57:59,840 INFO  org.apache.kafka.common.security.authenticator.AbstractLogin  - Successfully logged in.
2019-09-02 02:57:59,841 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh thread started.
2019-09-02 02:57:59,842 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
2019-09-02 02:57:59,919 WARN  org.apache.kafka.clients.consumer.ConsumerConfig              - The configuration 'zookeeper.connect' was supplied but isn't a known config.
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.10.2.0
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 576d93a8dc0cf421

Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version

to be 2.2.0 

"org.apache.kafka" % "kafka-clients" % "2.2.0" force() 

I also don't see 0.10.2.0 in the dependency tree of my build. 

Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?

Thanks,

Vishwas 

Thanks,

Vishwas