passing environment variables to flink program
Locked
passing environment variables to flink program
 
|
I am doing a Flink evaluation against Spark. My program is a JNI application and requires LD_LIBRARY_PATH to be set among other needed variables. In Spark I can do that multiple ways. One way is to use --conf 'spark.executorEnv.XXX=blah'. It works great.
How do I do that in Flink? thanks jackie |
Re: passing environment variables to flink program
|
|
Hi! What kind of setup are you using, YARN or standalone? In both modes, you should be able to pass your flags via the config entry "env.java.opts" in the flink-conf.yaml file. See here https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#other We have never passed library paths to link external JNI libraries with that, so let usknow if that works... Greetings, Stephan On Mon, Nov 2, 2015 at 10:12 AM, youarehow <[hidden email]> wrote: I am doing a Flink evaluation against Spark. My program is a JNI application |
Re: passing environment variables to flink program
|
|
This has less to do with JNI but much to do how to pass custom environment variables.
We are using YARN and the data is in HDFS. I have run the JNI program in local mode within Eclipse with no problem since I can set up the environment variables easily by using run configurations. Just I don't know how to do it when running inside YARN. The JNI program relies on LD_LIBRARY_PATH since the native side dynamically load other libraries (all copied on each node). public final class NativeProcessor{ static { System.loadLibrary("nativeprocessor");//environment variables must have been set before this!! } ... } For example, when loading libnativeprocessor.so, the so may use LD_LIBRARY_PATH or other custom environment variables to initialize, say, some static native variables. The env.java.opts can be used by java but not native libraries - especially in the above case it is used in a classloading static block so the variables need to be ready before loading native library. In hadoop mapreduce job we can use -Dmapreduce.map.env and -Dmapreduce.reduce.env to do this. In Spark we can use --conf 'spark.executor.XXX=blah'. I just can not find an equivalent in Flink yet. thanks jackie |
Re: passing environment variables to flink program
|
|
Ah, okay, I confused the issue. The environment variables would need to be defined or exported in the environment that spawns TaskManager processes. I think there is nothing for that in Flink yet, but it should not be hard to add that. Can you open an issue for that in JIRA? Thanks, Stephan On Mon, Nov 2, 2015 at 1:03 PM, Jian Jiang <[hidden email]> wrote: This has less to do with JNI but much to do how to pass custom environment |
RE: [IE] Re: passing environment variables to flink program
|
|
1.
Sure. I have created
FLINK-2954.
Thanks jackie From: [hidden email] [mailto:[hidden email]]
On Behalf Of Stephan Ewen Ah, okay, I confused the issue. The environment variables would need to be defined or exported in the environment that spawns TaskManager processes. I think there is nothing for that in Flink yet, but it should not be hard to add that. Can you open an issue for that in JIRA? Thanks, Stephan On Mon, Nov 2, 2015 at 1:03 PM, Jian Jiang <[hidden email]> wrote: This has less to do with JNI but much to do how to pass custom environment Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. |
Re: [IE] Re: passing environment variables to flink program
|
|
Thanks! Let's see if we can get that feature added soon... On Mon, Nov 2, 2015 at 3:02 PM, Jian Jiang <[hidden email]> wrote:
|
Re: [IE] Re: passing environment variables to flink program
|
|
I have to continue the evaluation so I managed to patch a fix to my local build. In case someone cannot wait here is what I did:
1. In the flink-core's ConfigConstants.java add public static final String FLINK_JVM_OPTIONS = "env.java.opts"; <-- existing public static final String FLINK_CUSTOM_ENV_VARS = "env.custom.env.vars"; 2. In the flink-conf.yaml now you can add environment variables separated by "|": env.custom.env.vars: name1=value1|name2=value2|name3=value3 .... 3. In the flink-yarn's YarnJobManager.scala's createContainerLaunchContext() method add the following between the two existing lines: Utils.setupEnv(yarnConf, containerEnv) <-- existing val customEnvOpts = flinkConfiguration.getString(ConfigConstants.FLINK_CUSTOM_ENV_VARS, "") if(customEnvOpts!=null) { val variables = customEnvOpts.split("\\|"); if(variables!=null&&variables.length>0) { for(variable <- variables) { val pair = variable.split("="); containerEnv.put(pair(0), pair(1)); } } } containerEnv.put(FlinkYarnClientBase.ENV_CLIENT_USERNAME, yarnClientUsername) <-- existing Now the container running my program sees the env variables. Note this is temp work around for local personally and should be discarded once Flink 1.0 comes out. |
|
|
Just a little note, the feature requested in FLINK-2954 has been implemented and is available in 1.0-SNAPSHOT now. Please let me know if its working as expected. On Wed, Nov 4, 2015 at 6:35 PM, Jian Jiang <[hidden email]> wrote: I have to continue the evaluation so I managed to patch a fix to my local |
RE: [IE] Re: [IE] Re: passing environment variables to flink program
|
|
Thanks Robert –
I will update you when get time. jackie From: rmetzger0 [via Apache Flink User Mailing List archive.] [mailto:ml-node+[hidden email]]
Just a little note, the feature requested in FLINK-2954 has
been implemented and is available in 1.0-SNAPSHOT now. Please let me know if its working as expected. On Wed, Nov 4, 2015 at 6:35 PM, Jian Jiang <[hidden email]> wrote: I have to continue the evaluation so I managed to patch a fix to my local Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. If you reply to this email, your message will be added to the discussion below: To unsubscribe from passing environment variables to flink program,
click here. |
| Free forum by Nabble | Edit this page |