[Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

Posted by Vijayendra Yadav on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-KAFKA-KEYTAB-Kafkaconsumer-error-Kerberos-tp37277.html

Hi Team,

I am trying to stream data from kafkaconsumer using: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html

Here my KAFKA is Kerberos secured and SSL enabled.

I am running my Flink streaming in yarn-cluster on EMR 5.31.

I have tried to pass keytab/principal in following 2 Ways:

1) Passing as JVM property in Flink run Command.

/usr/lib/flink/bin/flink run

   -yt ${app_install_path}/conf/                                                 \

-Dsecurity.kerberos.login.use-ticket-cache=false                              \
-yDsecurity.kerberos.login.use-ticket-cache=false                             \
-Dsecurity.kerberos.login.keytab=${app_install_path}/conf/keytab  \
-yDsecurity.kerberos.login.keytab=${app_install_path}/conf/.keytab \
-Djava.security.krb5.conf=${app_install_path}/conf/krb5.conf                  \
-yDjava.security.krb5.conf=${app_install_path}/conf/krb5.conf                 \
-Dsecurity.kerberos.login.principal=[hidden email]                 \
-yDsecurity.kerberos.login.principal= [hidden email]                \
-Dsecurity.kerberos.login.contexts=Client,KafkaClient                         \
-yDsecurity.kerberos.login.contexts=Client,KafkaClient

Here, I am getting the following Error, it seems like KEYTAB Was not transported to the run environment and probably not found.

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config'

2) Passing from flink config:  /usr/lib/flink/conf/flink-conf.yaml

security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab:  ${app_install_path}/conf/keytab
security.kerberos.login.principal:  [hidden email]
security.kerberos.login.contexts: Client,KafkaClient

Here, I am getting the following Error,

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config

 

Could you please help find, what are probable causes and resolution?

Regards,

Vijay