How to get the location of keytab when using flink on yarn

[sanmutongzi]

Hi,

i am using  the " per-job YARN session " mode deploy flink job on yarn and my flink

version is 1.4.1.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-kerberos.html

My use case is the yarn cluster where the flink job running is not enabled the kerberos mode in core-site.xml ,but i am trying to connecting an HBase cluster which is enabled kerberos. So i have to use the  loginUserFromKeytab() method to init kerberos infomation before  init the HBase connection. 

 UserGroupInformation.loginUserFromKeytab(user, keytabLocation);

So how can i get the keytab location  in my user code  ,  or is there any better ideas to solve the HBase kerberos problem on a yarn not using kerberos mode.

                                                                                                                           THANKS

[Stefan Richter]

Hi,

maybe Aljoscha or Eron (both in CC) can help you with this problem, I think they might know best about the Kerberos security.

Best,

Stefan

Am 20.09.2018 um 11:20 schrieb 杨光 <[hidden email]>:

Hi,

i am using  the " per-job YARN session " mode deploy flink job on yarn and my flink

version is 1.4.1.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-kerberos.html

My use case is the yarn cluster where the flink job running is not enabled the kerberos mode in core-site.xml ,but i am trying to connecting an HBase cluster which is enabled kerberos. So i have to use the  loginUserFromKeytab() method to init kerberos infomation before  init the HBase connection. 

 UserGroupInformation.loginUserFromKeytab(user, keytabLocation);

So how can i get the keytab location  in my user code  ,  or is there any better ideas to solve the HBase kerberos problem on a yarn not using kerberos mode.

                                                                                                                           THANKS

[Aljoscha Krettek]

Hi,

if the YARN cluster does not have Kerberos enabled then Flink will not ship the keytab file to the worker nodes. This means that you have to make sure it is available at some location where your application can use it yourself. But this might have security risks.

I'm afraid I don't know a better option now, maybe Eron knows one.

On 20. Sep 2018, at 19:51, Stefan Richter <[hidden email]> wrote:

Hi,

maybe Aljoscha or Eron (both in CC) can help you with this problem, I think they might know best about the Kerberos security.

Best,

Stefan

Am 20.09.2018 um 11:20 schrieb 杨光 <[hidden email]>:

Hi,

i am using  the " per-job YARN session " mode deploy flink job on yarn and my flink

version is 1.4.1.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-kerberos.html

My use case is the yarn cluster where the flink job running is not enabled the kerberos mode in core-site.xml ,but i am trying to connecting an HBase cluster which is enabled kerberos. So i have to use the  loginUserFromKeytab() method to init kerberos infomation before  init the HBase connection. 

 UserGroupInformation.loginUserFromKeytab(user, keytabLocation);

So how can i get the keytab location  in my user code  ,  or is there any better ideas to solve the HBase kerberos problem on a yarn not using kerberos mode.

                                                                                                                           THANKS

[sanmutongzi]

Hi Aljoscha,
Sorry for my late response . According to my experience ， if the
flink-conf.yaml has set the "security.kerberos.login.keytab" and
"security.kerberos.login.contexts" with a kerberos file then yarn will
ship the keytab file to the TaskManager .
Also i can find the log like:
 " INFO  org.apache.flink.configuration.GlobalConfiguration            -
Loading configuration property: security.kerberos.login.keytab,
/data1/yarn/nm/usercache/hadoop/appcache/application_1537515506704_0007/container_e28_1537515506704_0007_01_000001/krb5.keytab"
in the TaskManager log.
My problem is that in the user code like map or sink function how can i get
the security.kerberos.login.keytab value  for login .

THANKS




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Rong Rong]

Hi

Just a quick thought on this:

You might be able to use delegation token to access HBase[1]. It might be a more secure way instead of distributing your keytab over to all the YARN nodes.

Hope this helps.

--

Rong

[1] https://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication 

On Mon, Sep 24, 2018 at 7:51 PM sanmutongzi <[hidden email]> wrote:

Hi Aljoscha,
Sorry for my late response . According to my experience ， if the
flink-conf.yaml has set the "security.kerberos.login.keytab" and
"security.kerberos.login.contexts" with a kerberos file then yarn will
ship the keytab file to the TaskManager .
Also i can find the log like:
 " INFO  org.apache.flink.configuration.GlobalConfiguration            -
Loading configuration property: security.kerberos.login.keytab,
/data1/yarn/nm/usercache/hadoop/appcache/application_1537515506704_0007/container_e28_1537515506704_0007_01_000001/krb5.keytab"
in the TaskManager log.
My problem is that in the user code like map or sink function how can i get
the security.kerberos.login.keytab value  for login .

THANKS




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[sanmutongzi]

Hi Aljoscha,
Sorry for my late response . According to my experience ， if the
flink-conf.yaml has set the "security.kerberos.login.keytab" and
"security.kerberos.login.contexts" with a kerberos file then yarn will
ship the keytab file to the TaskManager .
Also i can find the log like:
 " INFO  org.apache.flink.configuration.GlobalConfiguration            -
Loading configuration property: security.kerberos.login.keytab,
/data1/yarn/nm/usercache/hadoop/appcache/application_1537515506704_0007/container_e28_1537515506704_0007_01_000001/krb5.keytab"
in the TaskManager log.
My problem is that in the user code like map or sink function how can i get
the security.kerberos.login.keytab value  for login .

THANKS

Aljoscha Krettek <[hidden email]> 于2018年9月20日周四 下午9:44写道：

Hi,

if the YARN cluster does not have Kerberos enabled then Flink will not ship the keytab file to the worker nodes. This means that you have to make sure it is available at some location where your application can use it yourself. But this might have security risks.

I'm afraid I don't know a better option now, maybe Eron knows one.

On 20. Sep 2018, at 19:51, Stefan Richter <[hidden email]> wrote:

Hi,

maybe Aljoscha or Eron (both in CC) can help you with this problem, I think they might know best about the Kerberos security.

Best,

Stefan

Am 20.09.2018 um 11:20 schrieb 杨光 <[hidden email]>:

Hi,

i am using  the " per-job YARN session " mode deploy flink job on yarn and my flink

version is 1.4.1.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-kerberos.html

My use case is the yarn cluster where the flink job running is not enabled the kerberos mode in core-site.xml ,but i am trying to connecting an HBase cluster which is enabled kerberos. So i have to use the  loginUserFromKeytab() method to init kerberos infomation before  init the HBase connection. 

 UserGroupInformation.loginUserFromKeytab(user, keytabLocation);

So how can i get the keytab location  in my user code  ,  or is there any better ideas to solve the HBase kerberos problem on a yarn not using kerberos mode.

                                                                                                                           THANKS