Flink on Yarn delegation token renewal

[Theofilos Kakantousis]

Hi everyone,

I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about
delegation token renewal when running Flink on Yarn. Yarn demands
services to renew delegation tokens on their own and if not, Yarn jobs
will fail after one week.
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html

For example, Spark has implemented a solution for that
https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala

and people had reported issues running Flink in the past,
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html

What is the status of this issue? How can Flink on Yarn handle this
issue so that jobs do not fail due to token expiration?

Cheers,
Theo

[Stefan Richter]

Hi,

I think this problem tracked in this issue: https://issues.apache.org/jira/browse/FLINK-3670 . This means that the current master and upcoming release 1.2 should work correctly.

Best,

Stefan

Am 08.11.2016 um 10:25 schrieb Theofilos Kakantousis <[hidden email]>:

Hi everyone,

I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about delegation token renewal when running Flink on Yarn. Yarn demands services to renew delegation tokens on their own and if not, Yarn jobs will fail after one week.
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html

For example, Spark has implemented a solution for that
https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala

and people had reported issues running Flink in the past,
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html

What is the status of this issue? How can Flink on Yarn handle this issue so that jobs do not fail due to token expiration?

Cheers,
Theo

[Theofilos Kakantousis]

Thank you for the prompt reply Stefan!

Cheers,
Theo

On 2016-11-08 11:29, Stefan Richter wrote:

Hi,

I think this problem tracked in this issue: https://issues.apache.org/jira/browse/FLINK-3670 . This means that the current master and upcoming release 1.2 should work correctly.

Best,

Stefan

Am 08.11.2016 um 10:25 schrieb Theofilos Kakantousis <[hidden email]>:

Hi everyone,

I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about delegation token renewal when running Flink on Yarn. Yarn demands services to renew delegation tokens on their own and if not, Yarn jobs will fail after one week.
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html

For example, Spark has implemented a solution for that
https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala

and people had reported issues running Flink in the past,
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html

What is the status of this issue? How can Flink on Yarn handle this issue so that jobs do not fail due to token expiration?

Cheers,
Theo