(DEPRECATED) Apache Flink User Mailing List archive.

understanding task manager logs

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
Vishwas Siravara
Reply | Threaded
Open this post in threaded view
♦
♦
|

understanding task manager logs

Vishwas Siravara
Hi guys,
I am using flink 1.7.2 and my application consumes from a kafka topic and publish to another kafka topic which is in its own kafka environment running a different kafka version,. I am using FlinkKafkaConsumer010 from this dependency 
"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. 

In the task manager log I see these lines: 
2019-09-02 02:57:59,840 INFO  org.apache.kafka.common.security.authenticator.AbstractLogin  - Successfully logged in.
2019-09-02 02:57:59,841 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh thread started.
2019-09-02 02:57:59,842 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
2019-09-02 02:57:59,919 WARN  org.apache.kafka.clients.consumer.ConsumerConfig              - The configuration 'zookeeper.connect' was supplied but isn't a known config.
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.10.2.0
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 576d93a8dc0cf421
Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version
to be 2.2.0 
"org.apache.kafka" % "kafka-clients" % "2.2.0" force() 
I also don't see 0.10.2.0 in the dependency tree of my build. 
Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?

Thanks,
Vishwas 

Thanks,
Vishwas


Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: understanding task manager logs

Fabian Hueske-2
Hi Vishwas,

This is a log statement from Kafka [1].
Not sure how when AppInfoParser is created (the log message is written by the constructor).

For Kafka versions > 1.0, I'd recommend the universal connector [2].

Not sure how well it works if producers and consumers have different versions.
Maybe Gordon (in CC) has some experience with that.

Best, Fabian


Am Di., 3. Sept. 2019 um 04:04 Uhr schrieb Vishwas Siravara <[hidden email]>:
Hi guys,
I am using flink 1.7.2 and my application consumes from a kafka topic and publish to another kafka topic which is in its own kafka environment running a different kafka version,. I am using FlinkKafkaConsumer010 from this dependency 
"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. 

In the task manager log I see these lines: 
2019-09-02 02:57:59,840 INFO  org.apache.kafka.common.security.authenticator.AbstractLogin  - Successfully logged in.
2019-09-02 02:57:59,841 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh thread started.
2019-09-02 02:57:59,842 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
2019-09-02 02:57:59,919 WARN  org.apache.kafka.clients.consumer.ConsumerConfig              - The configuration 'zookeeper.connect' was supplied but isn't a known config.
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.10.2.0
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 576d93a8dc0cf421
Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version
to be 2.2.0 
"org.apache.kafka" % "kafka-clients" % "2.2.0" force() 
I also don't see 0.10.2.0 in the dependency tree of my build. 
Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?

Thanks,
Vishwas 

Thanks,
Vishwas


Vishwas Siravara
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: understanding task manager logs

Vishwas Siravara
Thanks, I'll check it out.

On Thu, Sep 5, 2019 at 5:22 AM Fabian Hueske <[hidden email]> wrote:
Hi Vishwas,

This is a log statement from Kafka [1].
Not sure how when AppInfoParser is created (the log message is written by the constructor).

For Kafka versions > 1.0, I'd recommend the universal connector [2].

Not sure how well it works if producers and consumers have different versions.
Maybe Gordon (in CC) has some experience with that.

Best, Fabian


Am Di., 3. Sept. 2019 um 04:04 Uhr schrieb Vishwas Siravara <[hidden email]>:
Hi guys,
I am using flink 1.7.2 and my application consumes from a kafka topic and publish to another kafka topic which is in its own kafka environment running a different kafka version,. I am using FlinkKafkaConsumer010 from this dependency 
"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. 

In the task manager log I see these lines: 
2019-09-02 02:57:59,840 INFO  org.apache.kafka.common.security.authenticator.AbstractLogin  - Successfully logged in.
2019-09-02 02:57:59,841 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh thread started.
2019-09-02 02:57:59,842 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO  org.apache.kafka.common.security.kerberos.KerberosLogin       - [Principal=kafka/[hidden email]]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
2019-09-02 02:57:59,919 WARN  org.apache.kafka.clients.consumer.ConsumerConfig              - The configuration 'zookeeper.connect' was supplied but isn't a known config.
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.10.2.0
2019-09-02 02:57:59,919 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 576d93a8dc0cf421
Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version
to be 2.2.0 
"org.apache.kafka" % "kafka-clients" % "2.2.0" force() 
I also don't see 0.10.2.0 in the dependency tree of my build. 
Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?

Thanks,
Vishwas 

Thanks,
Vishwas


Free forum by Nabble Edit this page