Flink solution for having shared variable between task managers

[Soheil Pourbafrani]

Hi,

According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data.

What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance.

What is the standard way to achieve this? Does Flink provide any utility for that?

[Martin, Nick-2]

I think you’re looking for Broadcast State. Here’s a detailed guide.

 

https://flink.apache.org/2019/06/26/broadcast-state.html

 

From: Soheil Pourbafrani [mailto:[hidden email]]
Sent: Friday, January 17, 2020 6:50 AM
To: user <[hidden email]>
Subject: EXT :Flink solution for having shared variable between task managers

 

Hi,

 

According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data.

 

What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance.

 

What is the standard way to achieve this? Does Flink provide any utility for that?

[Fabian Hueske-2]

Hi,

I think you are looking for BroadcastState [1].

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html

Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani <[hidden email]>:

Hi,

According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data.

What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance.

What is the standard way to achieve this? Does Flink provide any utility for that?

[Soheil Pourbafrani]

Thanks, I'll check it out. 

On Mon, Feb 3, 2020 at 10:05 AM Fabian Hueske <[hidden email]> wrote:

Hi,

I think you are looking for BroadcastState [1].

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html

Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani <[hidden email]>:

Hi,

According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data.

What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance.

What is the standard way to achieve this? Does Flink provide any utility for that?