Including option for starting job and task managers in the foreground
Locked
Including option for starting job and task managers in the foreground
 
|
Hi All, Thanks, Brian |
Re: Including option for starting job and task managers in the foreground
|
|
Hi Brian, as far as I know this is at the moment not possible with our scripts. However it should be relatively easy to add by simply executing the Java command in flink-daemon.sh in the foreground. Do you want to add this? Cheers, On Dec 1, 2015 9:40 PM, "Brian Chhun" <[hidden email]> wrote:
|
Re: Including option for starting job and task managers in the foreground
|
|
Hi Brian, I don't recall Docker requires commands to run in the foreground. Still, if that is your requirement, simply remove the "&" at the end of this line in flink-daemon.sh: $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null & Cheers, Max On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <[hidden email]> wrote:
|
Re: Including option for starting job and task managers in the foreground
|
|
Yep, I think this makes sense. I'm currently patching the flink-daemon.sh script to remove the `&`, but I don't think it's a very robust solution, particularly when this script changes across versions of Flink. I'm very new to Docker, but the resources I've found indicates that the process must run in the foreground, though people seem to get around it with some hacks. When I have some time, I can look into refactoring some parts of the scripts so that it can be started in the foreground. Thanks, Brian On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <[hidden email]> wrote:
|
Re: Including option for starting job and task managers in the foreground
|
|
Have you looked at
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink ? This demonstrates how to use Flink with Docker. In particular it states: "Images [..] run Supervisor to stay alive when running containers." Have a look at flink/config-flink.sh. Cheers, Max On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun <[hidden email]> wrote: > Yep, I think this makes sense. I'm currently patching the flink-daemon.sh > script to remove the `&`, but I don't think it's a very robust solution, > particularly when this script changes across versions of Flink. I'm very new > to Docker, but the resources I've found indicates that the process must run > in the foreground, though people seem to get around it with some hacks. > > When I have some time, I can look into refactoring some parts of the scripts > so that it can be started in the foreground. > > Thanks, > Brian > > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <[hidden email]> wrote: >> >> Hi Brian, >> >> I don't recall Docker requires commands to run in the foreground. Still, >> if that is your requirement, simply remove the "&" at the end of this line >> in flink-daemon.sh: >> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null & >> >> Cheers, >> Max >> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <[hidden email]> >> wrote: >>> >>> Hi Brian, >>> >>> as far as I know this is at the moment not possible with our scripts. >>> However it should be relatively easy to add by simply executing the Java >>> command in flink-daemon.sh in the foreground. Do you want to add this? >>> >>> Cheers, >>> Till >>> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <[hidden email]> >>> wrote: >>>> >>>> Hi All, >>>> >>>> Is it possible to include a command line flag for starting job and task >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these >>>> things in the background. I'd like to execute these commands inside a docker >>>> container, but it's expected that the process is running in the foreground. >>>> I think it might be useful to have it run in the foreground so that it can >>>> be hooked into some process supervisors. Any suggestions are appreciated. >>>> >>>> >>>> Thanks, >>>> Brian >> >> > |
Re: Including option for starting job and task managers in the foreground
|
|
Thanks, I'm basing the things I'm doing based on what I see there. One thing that's not clear to me in that example is why supervisor is used to keep the container alive, rather than using some simpler means. It doesn't look like it's been configured to supervise anything. On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <[hidden email]> wrote: Have you looked at |
Re: Including option for starting job and task managers in the foreground
|
|
I think the way supervisor is used in the Docker scripts is a bit hacky. It is simply started in the foreground and does nothing. Supervisor is actually a really nice utility to start processes in Docker containers and monitor them. Cheers,Nevertheless, supervisor also expects commands to stay in the foreground. A common way to work around this, is to create a script which monitors the daemon process' pid. Thinking about this, I think we could actually add the foreground functionality directly in the jobmanager / taskmanager shell script like you suggested. In the meantime, you could also use a simple script like this: #!/usr/bin/env bash # daemonize job manager ./bin/jobmanager start cluster # wait until process goes down wait $! Max On Wed, Dec 2, 2015 at 7:16 PM, Brian Chhun <[hidden email]> wrote:
|
Re: Including option for starting job and task managers in the foreground
|
|
Thanks Max, I took a look at making this change directly to the scripts. I was initially thinking about making a separate script whose only responsibility is to run the command in the foreground, so that the flink-daemon.sh could delegate to this script. I didn't get very far into though, mostly trying to find a nice way to share the logging configuration and passing of parameters. While doing this, it did occur to me that if the process is run in the foreground, the logging should be written to standard out. This ended up being slightly hairy, at least in the job manager case, because the web UI client expects the standard out of the job manager to be written to a file, but I seem to have gotten things to work. I just wanted to mention this detail in case it's the convention for things running in the foreground, in which case it may need to be implemented alongside foregrounding the process. Thanks, Brian On Thu, Dec 3, 2015 at 3:55 AM, Maximilian Michels <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |