multiple users per flink deployment

[Georg Heiler]

Hi,

flink currently only seems to support a single kerberos ticket for deployment. Are there plans to support different users per each job?

regards,

Georg

[Tzu-Li (Gordon) Tai]

Hi,

There’s been quite a few requests on this recently on the mailing lists and also mentioned by some users offline, so I think we may need to start with plans to probably support this.

I’m CC’ing Eron to this thread to see if he has any thoughts on this, as he was among the first authors driving the Kerberos support in Flink.

I’m not really sure if such a feature support makes sense, given that all jobs of a single Flink deployment have full privileges and therefore no isolation in between.

Related question: what external service are you trying to authenticate to with different users?

If it is Kafka and perhaps you have different users for the consumer / producer, that will be very soon available in 1.3.2, which includes a version bump to Kafka 0.10 that allows multiple independent users within the same JVM through dynamic JAAS configuration.

See this mail thread [1] for more detail on that.

Cheers,

Gordon

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-0-10-jaas-multiple-clients-td12831.html#a13317

On 1 August 2017 at 6:16:08 PM, Georg Heiler ([hidden email]) wrote:

Hi,

flink currently only seems to support a single kerberos ticket for deployment. Are there plans to support different users per each job?

regards,

Georg

[Georg Heiler]

Thanks for the overview.
Currently a single flink cluster seems to run all tasks with the same user. I would want to be able to run each flink job as a separate user instead.

The update for separate read/write users is nice though.

Tzu-Li (Gordon) Tai <[hidden email]> schrieb am Mi. 2. Aug. 2017 um 10:59:

Hi,

There’s been quite a few requests on this recently on the mailing lists and also mentioned by some users offline, so I think we may need to start with plans to probably support this.

I’m CC’ing Eron to this thread to see if he has any thoughts on this, as he was among the first authors driving the Kerberos support in Flink.

I’m not really sure if such a feature support makes sense, given that all jobs of a single Flink deployment have full privileges and therefore no isolation in between.

Related question: what external service are you trying to authenticate to with different users?

If it is Kafka and perhaps you have different users for the consumer / producer, that will be very soon available in 1.3.2, which includes a version bump to Kafka 0.10 that allows multiple independent users within the same JVM through dynamic JAAS configuration.

See this mail thread [1] for more detail on that.

Cheers,

Gordon

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-0-10-jaas-multiple-clients-td12831.html#a13317

On 1 August 2017 at 6:16:08 PM, Georg Heiler ([hidden email]) wrote:

Hi,

flink currently only seems to support a single kerberos ticket for deployment. Are there plans to support different users per each job?

regards,

Georg

[Eron Wright]

One of the key challenges is isolation, eg. ensuring that one job cannot access the credentials of another.  The easiest solution today is to use the YARN deployment mode, with a separate app per job.  Meanwhile, improvements being made under the FLIP-6 banner for 1.4+ are lying groundwork for a multiuser experience. 

Hope this helps! 

On Aug 2, 2017 8:29 AM, "Georg Heiler" <[hidden email]> wrote:

Thanks for the overview.
Currently a single flink cluster seems to run all tasks with the same user. I would want to be able to run each flink job as a separate user instead.

The update for separate read/write users is nice though.

Tzu-Li (Gordon) Tai <[hidden email]> schrieb am Mi. 2. Aug. 2017 um 10:59:

Hi,

There’s been quite a few requests on this recently on the mailing lists and also mentioned by some users offline, so I think we may need to start with plans to probably support this.

I’m CC’ing Eron to this thread to see if he has any thoughts on this, as he was among the first authors driving the Kerberos support in Flink.

I’m not really sure if such a feature support makes sense, given that all jobs of a single Flink deployment have full privileges and therefore no isolation in between.

Related question: what external service are you trying to authenticate to with different users?

If it is Kafka and perhaps you have different users for the consumer / producer, that will be very soon available in 1.3.2, which includes a version bump to Kafka 0.10 that allows multiple independent users within the same JVM through dynamic JAAS configuration.

See this mail thread [1] for more detail on that.

Cheers,

Gordon

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-0-10-jaas-multiple-clients-td12831.html#a13317

On 1 August 2017 at 6:16:08 PM, Georg Heiler ([hidden email]) wrote:

Hi,

flink currently only seems to support a single kerberos ticket for deployment. Are there plans to support different users per each job?

regards,

Georg