How to set UserGroupInformation?

[Xinyu Zhang]

Hi all

I'm trying to write data to HDFS in directory "/home/hbase/XXX". Only "hbase" user can write to the directory.    

The problem is: I submit a job to yarn. The job will be a "yarn" user to write data, while "yarn" user is not allowed to write to  "/home/hbase/XXX".

Is there any method that I can set UserGroupInformation to "hbase"? 

Thanks!

Xinyu Zhang

[Till Rohrmann]

Hi,

have you tried exporting `HADOOP_USER_NAME` with the hbase user before running your application?

Cheers,

Till

On Fri, Jun 1, 2018 at 6:31 AM, Xinyu Zhang <[hidden email]> wrote:

Hi all

I'm trying to write data to HDFS in directory "/home/hbase/XXX". Only "hbase" user can write to the directory.    

The problem is: I submit a job to yarn. The job will be a "yarn" user to write data, while "yarn" user is not allowed to write to  "/home/hbase/XXX".

Is there any method that I can set UserGroupInformation to "hbase"? 

Thanks!

Xinyu Zhang

[Rong Rong]

Hi Xinyu,

To add to Till's comment, setting `HADOOP_USER_NAME` in your environment is probably the easiest way if you are using CLI.

If you are launching the job programmatically, e.g. using YarnClusterDescriptor [1], there're many ways to set `HADOOP_USER_NAME` as well, please share more information if you are going down that path.

Alternatively if you are set to use UserGroupInformation and "yarn" is super user in your cluster, you can also try out proxy user approach [2].

Hope this helps.

Thanks,

Rong

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/org/apache/flink/yarn/YarnClusterDescriptor.html

[2] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html

On Fri, Jun 1, 2018 at 7:25 AM, Till Rohrmann <[hidden email]> wrote:

Hi,

have you tried exporting `HADOOP_USER_NAME` with the hbase user before running your application?

Cheers,

Till

On Fri, Jun 1, 2018 at 6:31 AM, Xinyu Zhang <[hidden email]> wrote:

Hi all

I'm trying to write data to HDFS in directory "/home/hbase/XXX". Only "hbase" user can write to the directory.    

The problem is: I submit a job to yarn. The job will be a "yarn" user to write data, while "yarn" user is not allowed to write to  "/home/hbase/XXX".

Is there any method that I can set UserGroupInformation to "hbase"? 

Thanks!

Xinyu Zhang

[Xinyu Zhang]

Thanks both of you. Our hdfs version is 1.x, so I have to set hadoop.job.ugi. However, I can't find anywhere I can set the config. 

Do you have any ideas?

In fact, except

2018年6月2日星期六，Rong Rong <[hidden email]> 写道：

Hi Xinyu,

To add to Till's comment, setting `HADOOP_USER_NAME` in your environment is probably the easiest way if you are using CLI.

If you are launching the job programmatically, e.g. using YarnClusterDescriptor [1], there're many ways to set `HADOOP_USER_NAME` as well, please share more information if you are going down that path.

Alternatively if you are set to use UserGroupInformation and "yarn" is super user in your cluster, you can also try out proxy user approach [2].

Hope this helps.

Thanks,

Rong

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/org/apache/flink/yarn/YarnClusterDescriptor.html

[2] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html

On Fri, Jun 1, 2018 at 7:25 AM, Till Rohrmann <[hidden email]> wrote:

Hi,

have you tried exporting `HADOOP_USER_NAME` with the hbase user before running your application?

Cheers,

Till

On Fri, Jun 1, 2018 at 6:31 AM, Xinyu Zhang <[hidden email]> wrote:

Hi all

I'm trying to write data to HDFS in directory "/home/hbase/XXX". Only "hbase" user can write to the directory.    

The problem is: I submit a job to yarn. The job will be a "yarn" user to write data, while "yarn" user is not allowed to write to  "/home/hbase/XXX".

Is there any method that I can set UserGroupInformation to "hbase"? 

Thanks!

Xinyu Zhang