Flink, Yarn and MapR Kerberos issue

[ani.desh1512]

I am trying to setup Flink 1.2 using yarn on MapR (v5.2.0). The MapR cluster, on which, I am trying to setup this is a secure cluster. But, this cluster does not use Kerberos. Mapr, by default, uses some variant of ssl and MapR also normally has its own JAAS .conf file, which it relies on.

When I try to run yarn-session.sh, I get the following error:

java.lang.RuntimeException: Hadoop security is enabled but the login user does not have Kerberos credentials
To resolve this I tried the following two things:

1. I had seen a somewhat similar mention of this issue on JIRA. The issue says that its resolved in 1.2 but the comments on that issue do not indicate that.
By the way, I have added "-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf" in the yarn-session.sh file. But I still the get the same issue.

So, is this issue resolved? What am I missing here? Why does Flink require Kerberos credentials when MapR has no Kerberos setup?

2. I also tried specifying following in flink-conf.yaml:
security.ssl.enabled: true
security.ssl.keystore: /opt/mapr/conf/ssl_keystore
security.ssl.keystore-password: <>
security.ssl.key-password: <>
security.ssl.truststore: /opt/mapr/conf/ssl_truststore
security.ssl.truststore-password: <>

But, this too did not solve the problem and I get the same issue. Why is Flink trying to get Kerberos credentials even after ssl security is enabled?

Thanks,
Aniket

[Tzu-Li (Gordon) Tai]

Hi Aniket,

Thanks a lot for reporting this.

I’m afraid this seems to be a bug with Flink on YARN’s Kerberos authentication. It is incorrectly checking for Kerberos credentials even for non-Kerberos authentication methods.

I’ve filed a JIRA for this: https://issues.apache.org/jira/browse/FLINK-5949.

For the time being, I don’t think there’s a simple way to workaround it before the bug is fixed, because the bug indicates that whatever security type is enabled, Kerberos is used. We should probably have this fixed soon in the next bug fix release for Flink 1.2.

- Gordon

On March 2, 2017 at 7:11:02 AM, ani.desh1512 ([hidden email]) wrote:

I am trying to setup Flink 1.2 using yarn on MapR (v5.2.0). The MapR cluster,
on which, I am trying to setup this is a secure cluster. But, this cluster
does not use Kerberos. Mapr, by default, uses some variant of ssl
<http://maprdocs.mapr.com/home/SecurityGuide/Enable-wire-level-security.html>
and MapR also normally has its own JAAS .conf file, which it relies on.

When I try to run yarn-session.sh, I get the following error:

/java.lang.RuntimeException: Hadoop security is enabled but the login user
does not have Kerberos credentials/
To resolve this I tried the following two things:

1. I had seen a somewhat similar mention of this issue on JIRA
<https://issues.apache.org/jira/browse/FLINK-5055> . The issue says that
its resolved in 1.2 but the comments on that issue do not indicate that.
By the way, I have added
"-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf" in the
yarn-session.sh file. But I still the get the same issue.

So, is this issue resolved? What am I missing here? Why does Flink require
Kerberos credentials when MapR has no Kerberos setup?

2. I also tried specifying following in flink-conf.yaml:
security.ssl.enabled: true
security.ssl.keystore: /opt/mapr/conf/ssl_keystore
security.ssl.keystore-password: <>
security.ssl.key-password: <>
security.ssl.truststore: /opt/mapr/conf/ssl_truststore
security.ssl.truststore-password: <>

But, this too did not solve the problem and I get the same issue. Why is
Flink trying to get Kerberos credentials even after ssl security is enabled?

Thanks,
Aniket





--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Yarn-and-MapR-Kerberos-issue-tp11996.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[ani.desh1512]

So, I was able to circumvent this issue. This is in no way a permanent solution, but I thought I should let you (and anybody who encounters this problem in future) know some of my observations.
What I fount out was that,
1. In Mapr's version of hadoop, they do the authentication inside initialize() method of UserGroupInformation.java.
2. So, we would not need to check for kerberos credentials in flink's deploy() method of AbstractYarnClusterDescriptor.java (atleast for MapR's Hadoop version).
3. Also, the authentication method returned by MapR (via getAuthenticationMethod()) is CUSTOM.
4. I added a check for authenticationMethod, so that flink will check for hasKerberosAuthentication() ONLY if
the authentication method is Kerberos.
5. After doing this change, and building flink, I was able to confirm that indeed a user with appropriate MapR credentials was able to login without issues and an error was raised for an user without credentials. This is the desired behavior that we wanted.

 

[Tzu-Li (Gordon) Tai]

Hi Aniket!

Thanks for also looking into the problem!

I think checking `getAuthenticationMethod` on the UGI subject is the way to go.

At the moment I don’t think there’s a better “proper” solution for this.

As explained in the JIRA, we simply should not be checking for Kerberos credentials for all kinds of authentication methods, just Kerberos.

I’m currently doing some final tests on a MapR Sandbox with the bug fix PR, will open it very soon.

Would be great if you could take a look at the proposed fix too (it’s basically following along the lines you mentioned here :-) ).

Cheers,

Gordon

On March 14, 2017 at 6:10:43 AM, ani.desh1512 ([hidden email]) wrote:

So, I was able to circumvent this issue. This is in no way a permanent
solution, but I thought I should let you (and anybody who encounters this
problem in future) know some of my observations.
What I fount out was that,
1. In Mapr's version of hadoop, they do the authentication inside
initialize() method of UserGroupInformation.java.
2. So, we would not need to check for kerberos credentials in flink's
deploy() method of AbstractYarnClusterDescriptor.java (atleast for MapR's
Hadoop version).
3. Also, the authentication method returned by MapR (via
getAuthenticationMethod()) is CUSTOM.
4. I added a check for authenticationMethod, so that flink will check for
hasKerberosAuthentication() ONLY if
the authentication method is Kerberos.
5. After doing this change, and building flink, I was able to confirm that
indeed a user with appropriate MapR credentials was able to login without
issues and an error was raised for an user without credentials. This is the
desired behavior that we wanted.





--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Yarn-and-MapR-Kerberos-issue-tp11996p12194.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.