Creating singleton objects per task manager

[Salva Alcántara]

I need to create a singleton (manager) object to be used within all the
parallel instances of my UDF operator (a `ProcessFunction`). What is the
proper way of creating such a singleton object per task manager?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Seth Wiesman]

Hi Salva,

One TaskManager == One JVM. There is nothing Flink specific here, you can just create a singleton how you would in any other JVM application. But be careful, if your singleton does any sort of locking/coordination it will quickly become the bottleneck in your application. I would strongly encourage you to create one instance of your object per ProcessFunction, inside of open. That would be one instance per slot which is not equal to the parallelism of your operator. This would not create an object per key if that is your concern.

On Mon, Apr 6, 2020 at 5:26 AM Salva Alcántara <[hidden email]> wrote:

I need to create a singleton (manager) object to be used within all the
parallel instances of my UDF operator (a `ProcessFunction`). What is the
proper way of creating such a singleton object per task manager?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Salva Alcántara]

Thanks a lot Seth!



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[KristoffSC]

Hi Seth,
I would like to piggyback on this question :)

You wrote:
"I would strongly encourage you to create one instance of your object per
ProcessFunction, inside of open. That would be one instance per slot which
is not equal to the parallelism of your operator."

Especially the second part "That would be one instance per slot which is not
equal to the parallelism of your operator"

For my understanding the number of processFunction instances is equal to the
parallelism level of this operator. Paralleled instances are not deployed on
the same task slot, therefore if you create objects in open() method then
you will have as many objects as there are processFunction instances which
in my understanding is equal to the parallelism level for this operator.

Thanks,



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Seth Wiesman]

Hi Kristoff,

You are correct that, that was a typo :)
At most one instance per slot.

Seth  

> On Apr 7, 2020, at 9:41 AM, KristoffSC <[hidden email]> wrote:
>
> Hi Seth,
> I would like to piggyback on this question :)
>
> You wrote:
> "I would strongly encourage you to create one instance of your object per
> ProcessFunction, inside of open. That would be one instance per slot which
> is not equal to the parallelism of your operator."
>
> Especially the second part "That would be one instance per slot which is not
> equal to the parallelism of your operator"
>
> For my understanding the number of processFunction instances is equal to the
> parallelism level of this operator. Paralleled instances are not deployed on
> the same task slot, therefore if you create objects in open() method then
> you will have as many objects as there are processFunction instances which
> in my understanding is equal to the parallelism level for this operator.
>
> Thanks,
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/