Re: Flink job on secure Yarn fails after many hours

Posted by jimdowling on Nov 03, 2016; 8:31pm
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-tp3856p9888.html

Hi
Has anything ever happened on this issue, and not will it be addressed for 1.2?
It's a blocker for us.

To quote the YARN security docs:
"Any YARN service intended to run for an extended period of time must have a strategy for renewing credentials."

Reference:
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html

Spark has this implemented as a thread in its application master that periodically renews delegation tokens with (1) hdfs and (2) yarn:
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_yarn_long_jobs.html