Issues with forwarding environment variables
Locked
Issues with forwarding environment variables
 
|
Hi Need to forward a few env variables to Job and Task manager. I am running jobs in Yarn cluster I was referring to this : Forwarding I also found Stack Overflow I was able to configure and see the variables in Flink Dashboard But the task manager logs stills says `The system environment variable SERVICE_NAME is missing` as an exception message. The code trying to fetch it is as follows private String serviceName = System.getenv("SERVICE_NAME"); Is the fetched one not the same as set one ? How to set / fetch environment variables in such case ? |
Re: Issues with forwarding environment variables
|
|
Hi,
Could you please share the relevant parts of your flink-conf.yaml? Regards, Roman On Thu, May 20, 2021 at 9:13 PM Milind Vaidya <[hidden email]> wrote: > > Hi > > Need to forward a few env variables to Job and Task manager. > I am running jobs in Yarn cluster > I was referring to this : Forwarding > > I also found Stack Overflow > > I was able to configure and see the variables in Flink Dashboard > > But the task manager logs stills says > > `The system environment variable SERVICE_NAME is missing` as an exception message. > > The code trying to fetch it is as follows > > private String serviceName = System.getenv("SERVICE_NAME"); > > Is the fetched one not the same as set one ? How to set / fetch environment variables in such case ? > |
Re: Issues with forwarding environment variables
|
|
Hi Roman, I have added following lines to conf/flink-conf.yaml containerized.taskmanager.env.SERVICE_NAME: "test_service_name" containerized.master.env.SERVICE_NAME: "test_service_name" On Thu, May 20, 2021 at 12:30 PM Roman Khachatryan <[hidden email]> wrote: Hi, |
Re: Issues with forwarding environment variables
|
|
Thanks, it should work. I've created a ticket to track the issue [1].
Could you please specify Flink and Yarn versions you are using? You can also use properties (which don't depend on Yarn integration), for example like this: In flink-conf.yaml: env.java.opts.taskmanager: -DSERVICE_NAME=... In the application: System.getProperty("SERVICE_NAME"); Regards, Roman On Thu, May 20, 2021 at 9:50 PM Milind Vaidya <[hidden email]> wrote: > > > Hi Roman, > > I have added following lines to conf/flink-conf.yaml > > containerized.taskmanager.env.SERVICE_NAME: "test_service_name" > containerized.master.env.SERVICE_NAME: "test_service_name" > > > > > > On Thu, May 20, 2021 at 12:30 PM Roman Khachatryan <[hidden email]> wrote: >> >> Hi, >> >> Could you please share the relevant parts of your flink-conf.yaml? >> >> Regards, >> Roman >> >> On Thu, May 20, 2021 at 9:13 PM Milind Vaidya <[hidden email]> wrote: >> > >> > Hi >> > >> > Need to forward a few env variables to Job and Task manager. >> > I am running jobs in Yarn cluster >> > I was referring to this : Forwarding >> > >> > I also found Stack Overflow >> > >> > I was able to configure and see the variables in Flink Dashboard >> > >> > But the task manager logs stills says >> > >> > `The system environment variable SERVICE_NAME is missing` as an exception message. >> > >> > The code trying to fetch it is as follows >> > >> > private String serviceName = System.getenv("SERVICE_NAME"); >> > >> > Is the fetched one not the same as set one ? How to set / fetch environment variables in such case ? >> > |
Re: Issues with forwarding environment variables
|
|
here are the entries from taskmanager logs 2021-05-20 13:34:13,739 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.taskmanager, "-DSERVICE_NAME=hello-test,-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" 2021-05-20 13:34:13,740 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region 2021-05-20 13:34:13,742 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.SERVICE_NAME, "hello-test" 2021-05-20 13:34:13,743 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.SERVICE_NAME, "hello-test" But the error still persists On Thu, May 20, 2021 at 1:20 PM Roman Khachatryan <[hidden email]> wrote: Thanks, it should work. I've created a ticket to track the issue [1]. |
Re: Issues with forwarding environment variables
|
|
> private String serviceName = System.getenv("SERVICE_NAME");
Is it a scala object? If so, it can be initialized before any properties are set. What happens if the variable/property is read later at run time? Regards, Roman On Thu, May 20, 2021 at 10:41 PM Milind Vaidya <[hidden email]> wrote: > > here are the entries from taskmanager logs > > 2021-05-20 13:34:13,739 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.taskmanager, "-DSERVICE_NAME=hello-test,-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" > 2021-05-20 13:34:13,740 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region > 2021-05-20 13:34:13,742 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.SERVICE_NAME, "hello-test" > 2021-05-20 13:34:13,743 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.SERVICE_NAME, "hello-test" > > But the error still persists > > > On Thu, May 20, 2021 at 1:20 PM Roman Khachatryan <[hidden email]> wrote: >> >> Thanks, it should work. I've created a ticket to track the issue [1]. >> Could you please specify Flink and Yarn versions you are using? >> >> You can also use properties (which don't depend on Yarn integration), >> for example like this: >> In flink-conf.yaml: env.java.opts.taskmanager: -DSERVICE_NAME=... >> In the application: System.getProperty("SERVICE_NAME"); >> >> Regards, >> Roman >> >> On Thu, May 20, 2021 at 9:50 PM Milind Vaidya <[hidden email]> wrote: >> > >> > >> > Hi Roman, >> > >> > I have added following lines to conf/flink-conf.yaml >> > >> > containerized.taskmanager.env.SERVICE_NAME: "test_service_name" >> > containerized.master.env.SERVICE_NAME: "test_service_name" >> > >> > >> > >> > >> > >> > On Thu, May 20, 2021 at 12:30 PM Roman Khachatryan <[hidden email]> wrote: >> >> >> >> Hi, >> >> >> >> Could you please share the relevant parts of your flink-conf.yaml? >> >> >> >> Regards, >> >> Roman >> >> >> >> On Thu, May 20, 2021 at 9:13 PM Milind Vaidya <[hidden email]> wrote: >> >> > >> >> > Hi >> >> > >> >> > Need to forward a few env variables to Job and Task manager. >> >> > I am running jobs in Yarn cluster >> >> > I was referring to this : Forwarding >> >> > >> >> > I also found Stack Overflow >> >> > >> >> > I was able to configure and see the variables in Flink Dashboard >> >> > >> >> > But the task manager logs stills says >> >> > >> >> > `The system environment variable SERVICE_NAME is missing` as an exception message. >> >> > >> >> > The code trying to fetch it is as follows >> >> > >> >> > private String serviceName = System.getenv("SERVICE_NAME"); >> >> > >> >> > Is the fetched one not the same as set one ? How to set / fetch environment variables in such case ? >> >> > |
Re: Issues with forwarding environment variables
|
|
This is java code. I have a flink job running and it is trying to fetch this variable at run time itself. I see the properties getting reflected in the logs as already mentioned but not visible from the code. On Thu, May 20, 2021 at 1:53 PM Roman Khachatryan <[hidden email]> wrote: > private String serviceName = System.getenv("SERVICE_NAME"); |
Re: Issues with forwarding environment variables
|
|
Hi, Milind
Could you help to provide the skeleton of your job code? Actually, if you implement a custom function, like Tokenizer in the WordCount example, the class member will be initialized at the client-side and be serialized to the task manager. As a result, neither the system envs nor the system properties at the TaskManager will be used. If that is the case, you can initiate the `serviceName` field in the map/flatMap or open function. Then, it will read the TM's envs or properties instead. Best, Yangze Guo On Fri, May 21, 2021 at 5:40 AM Milind Vaidya <[hidden email]> wrote: > > This is java code. I have a flink job running and it is trying to fetch this variable at run time itself. I see the properties getting reflected in the logs as already mentioned but not visible from the code. > > On Thu, May 20, 2021 at 1:53 PM Roman Khachatryan <[hidden email]> wrote: >> >> > private String serviceName = System.getenv("SERVICE_NAME"); >> Is it a scala object? If so, it can be initialized before any >> properties are set. >> What happens if the variable/property is read later at run time? >> >> Regards, >> Roman >> >> On Thu, May 20, 2021 at 10:41 PM Milind Vaidya <[hidden email]> wrote: >> > >> > here are the entries from taskmanager logs >> > >> > 2021-05-20 13:34:13,739 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.taskmanager, "-DSERVICE_NAME=hello-test,-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" >> > 2021-05-20 13:34:13,740 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region >> > 2021-05-20 13:34:13,742 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.SERVICE_NAME, "hello-test" >> > 2021-05-20 13:34:13,743 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.SERVICE_NAME, "hello-test" >> > >> > But the error still persists >> > >> > >> > On Thu, May 20, 2021 at 1:20 PM Roman Khachatryan <[hidden email]> wrote: >> >> >> >> Thanks, it should work. I've created a ticket to track the issue [1]. >> >> Could you please specify Flink and Yarn versions you are using? >> >> >> >> You can also use properties (which don't depend on Yarn integration), >> >> for example like this: >> >> In flink-conf.yaml: env.java.opts.taskmanager: -DSERVICE_NAME=... >> >> In the application: System.getProperty("SERVICE_NAME"); >> >> >> >> Regards, >> >> Roman >> >> >> >> On Thu, May 20, 2021 at 9:50 PM Milind Vaidya <[hidden email]> wrote: >> >> > >> >> > >> >> > Hi Roman, >> >> > >> >> > I have added following lines to conf/flink-conf.yaml >> >> > >> >> > containerized.taskmanager.env.SERVICE_NAME: "test_service_name" >> >> > containerized.master.env.SERVICE_NAME: "test_service_name" >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Thu, May 20, 2021 at 12:30 PM Roman Khachatryan <[hidden email]> wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> Could you please share the relevant parts of your flink-conf.yaml? >> >> >> >> >> >> Regards, >> >> >> Roman >> >> >> >> >> >> On Thu, May 20, 2021 at 9:13 PM Milind Vaidya <[hidden email]> wrote: >> >> >> > >> >> >> > Hi >> >> >> > >> >> >> > Need to forward a few env variables to Job and Task manager. >> >> >> > I am running jobs in Yarn cluster >> >> >> > I was referring to this : Forwarding >> >> >> > >> >> >> > I also found Stack Overflow >> >> >> > >> >> >> > I was able to configure and see the variables in Flink Dashboard >> >> >> > >> >> >> > But the task manager logs stills says >> >> >> > >> >> >> > `The system environment variable SERVICE_NAME is missing` as an exception message. >> >> >> > >> >> >> > The code trying to fetch it is as follows >> >> >> > >> >> >> > private String serviceName = System.getenv("SERVICE_NAME"); >> >> >> > >> >> >> > Is the fetched one not the same as set one ? How to set / fetch environment variables in such case ? >> >> >> > |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |