Flink Hive connector: hive-conf-dir supports hdfs URI, while hadoop-conf-dir supports local path only?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Flink Hive connector: hive-conf-dir supports hdfs URI, while hadoop-conf-dir supports local path only?

Yik San Chan
Hi community,


In my current setup, local dev env can access testing env. I would like to run Flink job on local dev env, while reading/writing data from/to testing env Hive.

This is what I do:

```
CREATE CATALOG hive WITH (
    'type' = 'hive',
    'hive-conf-dir' = 'hdfs://testhdp273/hive/conf'
)
```

However, I realizes I also need to specify a matching Hadoop classpath, therefore I want to also define `hadoop-conf-dir` that actually points to the hadoop classpath in testing env. However, as said in [docs](https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/hive/#hadoop-conf-dir):

> Path to Hadoop conf dir. Only local file system paths are supported. The recommended way to set Hadoop conf is via the HADOOP_CONF_DIR environment variable. Use the option only if environment variable doesn't work for you, e.g. if you want to configure each HiveCatalog separately.

I wonder why hadoop-conf-dir only supports local path, while hive-conf-dir supports any legit hdfs path? Any work around to this problem?

Any help? Thanks!

Best,
Yik San