Using shell environment variables

[Abhishek Jain]

Hi!

I am using a library that depends on a certain environment variable set (mandatorily). Now, I've exported this variable in my environment but somehow it's not being read by the task manager. Following is the exception I get when I try to run the job:

Caused by: com.example.MyCustomException: Env token is null
at com.example.AerospikeSink.open(AerospikeSink.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)

Here's the code that throws this exception:

@Override
public void open(Configuration config) throws Exception {
    if (System.getenv("API_TOKEN") == null) {
        throw new MyCustomException("Env token is null");
    }
}

My question: Is there an alternative to System.getenv() that I can use to access environment variables inside of flink task?

( P.S. I've only copied relevant code snippet to avoid confusion. I do intend to use API_TOKEN later on. )

--

[miki haiat]

Did you register your system environment parameter ?

You can find here several ways to use configuration data [1]

1.https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html

On Sat, Aug 24, 2019, 20:26 Abhishek Jain <[hidden email]> wrote:

Hi!

I am using a library that depends on a certain environment variable set (mandatorily). Now, I've exported this variable in my environment but somehow it's not being read by the task manager. Following is the exception I get when I try to run the job:

Caused by: com.example.MyCustomException: Env token is null
at com.example.AerospikeSink.open(AerospikeSink.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)

Here's the code that throws this exception:

@Override
public void open(Configuration config) throws Exception {
    if (System.getenv("API_TOKEN") == null) {
        throw new MyCustomException("Env token is null");
    }
}

My question: Is there an alternative to System.getenv() that I can use to access environment variables inside of flink task?

( P.S. I've only copied relevant code snippet to avoid confusion. I do intend to use API_TOKEN later on. )

--

[Abhishek Jain]

Hi Miki,

Thanks for your reply. ParameterTool will only help in making the value accessible through ParameterTool.get(). However, I need a way of accessing the value using "System.getenv" since the underlying library uses it so.

On Sat, 24 Aug 2019 at 23:04, miki haiat <[hidden email]> wrote:

Did you register your system environment parameter ?

You can find here several ways to use configuration data [1]

1.https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html

On Sat, Aug 24, 2019, 20:26 Abhishek Jain <[hidden email]> wrote:

Hi!

I am using a library that depends on a certain environment variable set (mandatorily). Now, I've exported this variable in my environment but somehow it's not being read by the task manager. Following is the exception I get when I try to run the job:

Caused by: com.example.MyCustomException: Env token is null
at com.example.AerospikeSink.open(AerospikeSink.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)

Here's the code that throws this exception:

@Override
public void open(Configuration config) throws Exception {
    if (System.getenv("API_TOKEN") == null) {
        throw new MyCustomException("Env token is null");
    }
}

My question: Is there an alternative to System.getenv() that I can use to access environment variables inside of flink task?

( P.S. I've only copied relevant code snippet to avoid confusion. I do intend to use API_TOKEN later on. )

--

--

[Zhu Zhu]

Hi Abhishek,

You need to export the environment variables on all the worker machines(not the machine to submit the job).

Alternatively, if you are submitting the job to a yarn cluster, you can use flink conf prefix "containerized.taskmanager.env." to add environment variables to Flink's task manager process.

For example for passing LD_LIBRARY_PATH as an env variable to the workers, set: containerized.taskmanager.env.LD_LIBRARY_PATH: "/usr/lib/native" in the flink-conf.yaml.

Thanks,

Zhu Zhu

Abhishek Jain <[hidden email]> 于2019年8月25日周日 上午2:48写道：

Hi Miki,

Thanks for your reply. ParameterTool will only help in making the value accessible through ParameterTool.get(). However, I need a way of accessing the value using "System.getenv" since the underlying library uses it so.

On Sat, 24 Aug 2019 at 23:04, miki haiat <[hidden email]> wrote:

Did you register your system environment parameter ?

You can find here several ways to use configuration data [1]

1.https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html

On Sat, Aug 24, 2019, 20:26 Abhishek Jain <[hidden email]> wrote:

Hi!

I am using a library that depends on a certain environment variable set (mandatorily). Now, I've exported this variable in my environment but somehow it's not being read by the task manager. Following is the exception I get when I try to run the job:

Caused by: com.example.MyCustomException: Env token is null
at com.example.AerospikeSink.open(AerospikeSink.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)

Here's the code that throws this exception:

@Override
public void open(Configuration config) throws Exception {
    if (System.getenv("API_TOKEN") == null) {
        throw new MyCustomException("Env token is null");
    }
}

My question: Is there an alternative to System.getenv() that I can use to access environment variables inside of flink task?

( P.S. I've only copied relevant code snippet to avoid confusion. I do intend to use API_TOKEN later on. )

--

--

[Vishwas Siravara]

You can also link at runtime by providing the path to the dylib by adding -Djava.library.path=<path to dylib> in jvm options in the task manager 

On Sat, Aug 24, 2019 at 9:11 PM Zhu Zhu <[hidden email]> wrote:

Hi Abhishek,

You need to export the environment variables on all the worker machines(not the machine to submit the job).

Alternatively, if you are submitting the job to a yarn cluster, you can use flink conf prefix "containerized.taskmanager.env." to add environment variables to Flink's task manager process.

For example for passing LD_LIBRARY_PATH as an env variable to the workers, set: containerized.taskmanager.env.LD_LIBRARY_PATH: "/usr/lib/native" in the flink-conf.yaml.

Thanks,

Zhu Zhu

Abhishek Jain <[hidden email]> 于2019年8月25日周日 上午2:48写道：

Hi Miki,

Thanks for your reply. ParameterTool will only help in making the value accessible through ParameterTool.get(). However, I need a way of accessing the value using "System.getenv" since the underlying library uses it so.

On Sat, 24 Aug 2019 at 23:04, miki haiat <[hidden email]> wrote:

Did you register your system environment parameter ?

You can find here several ways to use configuration data [1]

1.https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html

On Sat, Aug 24, 2019, 20:26 Abhishek Jain <[hidden email]> wrote:

Hi!

I am using a library that depends on a certain environment variable set (mandatorily). Now, I've exported this variable in my environment but somehow it's not being read by the task manager. Following is the exception I get when I try to run the job:

Caused by: com.example.MyCustomException: Env token is null
at com.example.AerospikeSink.open(AerospikeSink.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)

Here's the code that throws this exception:

@Override
public void open(Configuration config) throws Exception {
    if (System.getenv("API_TOKEN") == null) {
        throw new MyCustomException("Env token is null");
    }
}

My question: Is there an alternative to System.getenv() that I can use to access environment variables inside of flink task?

( P.S. I've only copied relevant code snippet to avoid confusion. I do intend to use API_TOKEN later on. )

--

--