Thread Locals in Flink Streaming Application

[Vinay Patil]

Hi,

I am updating the DTO in one operator , so is it safe to use Thread Local for it, I am taking care to clear it, 

My confusion here is : is it required to create a Thread Local variable in a streaming application, if the same piece of code is executed by different threads, does flink handles this ?

Can we make use of open method to avoid ThreadLocal by initializing the variable there, since the open will be called once per thread (correct me here)

Can you please explain this to make it clear

[Till Rohrmann]

Hi Vinay,

I think the answer depends on what you want to achieve. 

Subtasks of an operator are all executed by separate threads. Some of them run in the same JVM others not. So if you want to share data across subtasks being executed in the same process, you can use static variables bound to the function of this operator.

In case of chaining, multiple subtasks (all belonging to different operators, though) are executed by the same thread. Here you can use thread local variables to share data between these operators in a thread safe manner.

If you create variables/initialize fields in the open method of an operator, then this variable is bound to the specific subtask instance which runs the code.

I hope this helps you a little bit. But I'm not sure whether I got your question completely right.

Cheers,

Till

On Fri, Sep 16, 2016 at 5:50 PM, Vinay Patil <[hidden email]> wrote:

Hi,

I am updating the DTO in one operator , so is it safe to use Thread Local for it, I am taking care to clear it, 

My confusion here is : is it required to create a Thread Local variable in a streaming application, if the same piece of code is executed by different threads, does flink handles this ?

Can we make use of open method to avoid ThreadLocal by initializing the variable there, since the open will be called once per thread (correct me here)

Can you please explain this to make it clear