Re: Flink on YARN: delegation token expired prevent job restart

Posted by Kien Truong on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-on-YARN-delegation-token-expired-prevent-job-restart-tp39520p39528.html

Hi, 

Yes, I did. There're also logs about logging in using keytab successfully in both Job Manager and Task Manager. 

I found some YARN docs about token renewal on AM restart

> Therefore, to survive AM restart after token expiry, your AM has to get the NMs to localize the keytab or make no HDFS accesses until (somehow) a new token has been passed to them from a client.

Maybe Flink did access HDFS with an expired token, before switching to use the localized keytab ?

Regards,

Kien

On 17 Nov 2020 at 15:14, Yangze Guo <[hidden email]> wrote:

Hi, Kien,



Do you config the "security.kerberos.login.principal" and the

"security.kerberos.login.keytab" together? If you only set the keytab,

it will not take effect.



Best,

Yangze Guo



On Tue, Nov 17, 2020 at 3:03 PM Kien Truong <[hidden email]> wrote:

>

> Hi all,

>

> We are having an issue where Flink Application Master is unable to automatically restart Flink job after its delegation token has expired.

>

> We are using Flink 1.11 with YARN 3.1.1 in single job per yarn-cluster mode. We have also add valid keytab configuration and taskmanagers are able to login with keytabs correctly. However, it seems YARN Application Master still use delegation tokens instead of the keytab.

>

> Any idea how to resolve this would be much appreciated.

>

> Thanks

> Kien

>

>

>

>