no valid hadoop home directory can be found

[Lydia Ickler]

Hi all,

I get the following error message that no valid hadoop home directory can be found when trying to initialize the HBase configuration.

Where would I specify that path?

12:41:02,043 INFO  org.apache.flink.addons.hbase.TableInputFormat                - Initializing HBaseConfiguration

12:41:02,174 DEBUG org.apache.hadoop.util.Shell                                  - Failed to detect a valid hadoop home directory

java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.

        at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)

        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)

        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)

        at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)

        at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)

        at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:259)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)

        at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1816)

        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)

        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)

        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)

        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)

        at org.apache.flink.addons.hbase.TableInputFormat.createTable(TableInputFormat.java:89)

        at org.apache.flink.addons.hbase.TableInputFormat.configure(TableInputFormat.java:78)

        at org.apache.flink.runtime.operators.DataSourceTask.initInputFormat(DataSourceTask.java:273)

        at org.apache.flink.runtime.operators.DataSourceTask.registerInputOutput(DataSourceTask.java:79)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)

        at java.lang.Thread.run(Thread.java:745)

[Ufuk Celebi]

Either specify the env variable HADOOP_HOME or set the JVM property ‘hadoop.home.dir’ (-Dhadoop.home.dir=…)

– Ufuk

> On 23 Sep 2015, at 12:43, Lydia Ickler <[hidden email]> wrote:
>
> Hi all,
>
> I get the following error message that no valid hadoop home directory can be found when trying to initialize the HBase configuration.
> Where would I specify that path?
>
> 12:41:02,043 INFO  org.apache.flink.addons.hbase.TableInputFormat                - Initializing HBaseConfiguration
> 12:41:02,174 DEBUG org.apache.hadoop.util.Shell                                  - Failed to detect a valid hadoop home directory
> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>         at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
>         at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
>         at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
>         at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
>         at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
>         at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:259)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
>         at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1816)
>         at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
>         at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)
>         at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)
>         at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)
>         at org.apache.flink.addons.hbase.TableInputFormat.createTable(TableInputFormat.java:89)
>         at org.apache.flink.addons.hbase.TableInputFormat.configure(TableInputFormat.java:78)
>         at org.apache.flink.runtime.operators.DataSourceTask.initInputFormat(DataSourceTask.java:273)
>         at org.apache.flink.runtime.operators.DataSourceTask.registerInputOutput(DataSourceTask.java:79)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
>         at java.lang.Thread.run(Thread.java:745)
>

[Lydia Ickler]

Added the -Dhadoop.home.dir= to flink-conf.yaml but still get the same error.
Could it be that it somehow has to be added to HBase directly?

> Am 23.09.2015 um 13:27 schrieb Ufuk Celebi <[hidden email]>:
>
> Either specify the env variable HADOOP_HOME or set the JVM property ‘hadoop.home.dir’ (-Dhadoop.home.dir=…)
>
> – Ufuk
>
>> On 23 Sep 2015, at 12:43, Lydia Ickler <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I get the following error message that no valid hadoop home directory can be found when trying to initialize the HBase configuration.
>> Where would I specify that path?
>>
>> 12:41:02,043 INFO  org.apache.flink.addons.hbase.TableInputFormat                - Initializing HBaseConfiguration
>> 12:41:02,174 DEBUG org.apache.hadoop.util.Shell                                  - Failed to detect a valid hadoop home directory
>> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>>        at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
>>        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
>>        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
>>        at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
>>        at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
>>        at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:259)
>>        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)
>>        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
>>        at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1816)
>>        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)
>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)
>>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)
>>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)
>>        at org.apache.flink.addons.hbase.TableInputFormat.createTable(TableInputFormat.java:89)
>>        at org.apache.flink.addons.hbase.TableInputFormat.configure(TableInputFormat.java:78)
>>        at org.apache.flink.runtime.operators.DataSourceTask.initInputFormat(DataSourceTask.java:273)
>>        at org.apache.flink.runtime.operators.DataSourceTask.registerInputOutput(DataSourceTask.java:79)
>>        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
>>        at java.lang.Thread.run(Thread.java:745)
>>
>

[rmetzger0]

I don't think this is a critical issue. Its logged at DEBUG level.

I think you would have to specify the environment variable on all machines in your cluster. BUT as far I know, HADOOP_HOME has been deprecated in newer Hadoop (2.2+) versions.

On Wed, Sep 23, 2015 at 1:41 PM, Lydia Ickler <[hidden email]> wrote:

Added the -Dhadoop.home.dir= to flink-conf.yaml but still get the same error.
Could it be that it somehow has to be added to HBase directly?

> Am 23.09.2015 um 13:27 schrieb Ufuk Celebi <[hidden email]>:
>
> Either specify the env variable HADOOP_HOME or set the JVM property ‘hadoop.home.dir’ (-Dhadoop.home.dir=…)
>
> – Ufuk
>
>> On 23 Sep 2015, at 12:43, Lydia Ickler <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I get the following error message that no valid hadoop home directory can be found when trying to initialize the HBase configuration.
>> Where would I specify that path?
>>
>> 12:41:02,043 INFO  org.apache.flink.addons.hbase.TableInputFormat                - Initializing HBaseConfiguration
>> 12:41:02,174 DEBUG org.apache.hadoop.util.Shell                                  - Failed to detect a valid hadoop home directory
>> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>>        at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
>>        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
>>        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
>>        at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
>>        at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
>>        at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:259)
>>        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)
>>        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
>>        at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:43)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1816)
>>        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:82)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)
>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)
>>        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)
>>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)
>>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)
>>        at org.apache.flink.addons.hbase.TableInputFormat.createTable(TableInputFormat.java:89)
>>        at org.apache.flink.addons.hbase.TableInputFormat.configure(TableInputFormat.java:78)
>>        at org.apache.flink.runtime.operators.DataSourceTask.initInputFormat(DataSourceTask.java:273)
>>        at org.apache.flink.runtime.operators.DataSourceTask.registerInputOutput(DataSourceTask.java:79)
>>        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
>>        at java.lang.Thread.run(Thread.java:745)
>>
>