Re: yarn session: one JVM per task

Posted by Xintong Song on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/yarn-session-one-JVM-per-task-tp33014p33050.html

Depending on your Flink version, the '-n' option might not take effect. It is removed in the latest release, but before that there were a few versions where this option is neither removed nor taking effect.

Anyway, as long as you have multiple containers, I don't think there's a way to make some of the tasks scheduled to the same JVM. Not that I'm aware of.

On Mon, Feb 24, 2020 at 8:43 PM David Morin <[hidden email]> wrote:

Hi,

Thanks Xintong.
I've noticed than when I use yarn-session.sh with --slots (-s) parameter but without --container (-n) it creates one task/slot per taskmanager. Before with the both n and -s it was not the case.
I prefer to use only small container with only one task to scale my pipeline and of course to prevent from thread-safe issue
Do you think I cannot be confident on that behaviour ?

Regards,
David

On 2020/02/22 17:11:25, David Morin <[hidden email]> wrote:
> Hi,
> My app is based on a lib that is not thread safe (yet...).
> In waiting of the patch has been pushed, how can I be sure that my Sink that uses this lib is in one JVM ?
> Context: I use one Yarn session and send my Flink jobs to this session
>
> Regards,
> David
>