(DEPRECATED) Apache Flink User Mailing List archive.

Shared state between two process functions

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
jaswin.shah@outlook.com
Reply | Threaded
Open this post in threaded view
♦
♦
|

Shared state between two process functions

jaswin.shah@outlook.com
Hi,

Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink?

Thanks,
Jaswin
Yun Gao
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Shared state between two process functions

Yun Gao
Hi Jaswin,
   
   Currently the state belongs to single operators, thus it should be not possible to share states between different operators. Could you also share the original problem want to solve by sharing states ?

Best,
 Yun


------------------Original Mail ------------------
Sender:Jaswin Shah <[hidden email]>
Send Date:Sun Jun 14 18:57:54 2020
Subject:Shared state between two process functions
Hi,

Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink?

Thanks,
Jaswin
jaswin.shah@outlook.com
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Shared state between two process functions

jaswin.shah@outlook.com
Basically,  I have multiple processafunctions and they are doing some.computations based on some historical results and the historical events and results are common across the process functions due to which I have a lot of redundant processing in many process functions   so, I have been thinking of some shared state accessible between multiple keyedprocessfunctions.

From: Yun Gao <[hidden email]>
Sent: Monday, June 15, 2020 8:27:38 AM
To: Jaswin Shah <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re: Shared state between two process functions
 
Hi Jaswin,
   
   Currently the state belongs to single operators, thus it should be not possible to share states between different operators. Could you also share the original problem want to solve by sharing states ?

Best,
 Yun


------------------Original Mail ------------------
Sender:Jaswin Shah <[hidden email]>
Send Date:Sun Jun 14 18:57:54 2020
Subject:Shared state between two process functions
Hi,

Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink?

Thanks,
Jaswin
rmetzger0
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Shared state between two process functions

rmetzger0
Thanks for sharing some details on the use case: Are you able to move the common computation into one operator that runs before the ProcessFunctions, and you are sending the results there?
You can build quite advanced dataflow graphs with Flink to model your problem.

On Mon, Jun 15, 2020 at 9:01 AM Jaswin Shah <[hidden email]> wrote:
Basically,  I have multiple processafunctions and they are doing some.computations based on some historical results and the historical events and results are common across the process functions due to which I have a lot of redundant processing in many process functions   so, I have been thinking of some shared state accessible between multiple keyedprocessfunctions.

From: Yun Gao <[hidden email]>
Sent: Monday, June 15, 2020 8:27:38 AM
To: Jaswin Shah <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re: Shared state between two process functions
 
Hi Jaswin,
   
   Currently the state belongs to single operators, thus it should be not possible to share states between different operators. Could you also share the original problem want to solve by sharing states ?

Best,
 Yun


------------------Original Mail ------------------
Sender:Jaswin Shah <[hidden email]>
Send Date:Sun Jun 14 18:57:54 2020
Subject:Shared state between two process functions
Hi,

Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink?

Thanks,
Jaswin
Congxian Qiu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Shared state between two process functions

Congxian Qiu
Hi

Maybe you can take a look at broadcast state[1]


Robert Metzger <[hidden email]> 于2020年6月16日周二 上午2:18写道：
Thanks for sharing some details on the use case: Are you able to move the common computation into one operator that runs before the ProcessFunctions, and you are sending the results there?
You can build quite advanced dataflow graphs with Flink to model your problem.

On Mon, Jun 15, 2020 at 9:01 AM Jaswin Shah <[hidden email]> wrote:
Basically,  I have multiple processafunctions and they are doing some.computations based on some historical results and the historical events and results are common across the process functions due to which I have a lot of redundant processing in many process functions   so, I have been thinking of some shared state accessible between multiple keyedprocessfunctions.

From: Yun Gao <[hidden email]>
Sent: Monday, June 15, 2020 8:27:38 AM
To: Jaswin Shah <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re: Shared state between two process functions
 
Hi Jaswin,
   
   Currently the state belongs to single operators, thus it should be not possible to share states between different operators. Could you also share the original problem want to solve by sharing states ?

Best,
 Yun


------------------Original Mail ------------------
Sender:Jaswin Shah <[hidden email]>
Send Date:Sun Jun 14 18:57:54 2020
Subject:Shared state between two process functions
Hi,

Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink?

Thanks,
Jaswin
Free forum by Nabble Edit this page