Making broadcast state queryable?

[Oytun Tez]

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

[Yu Li]

Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/r8k8a

On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

[Oytun Tez]

Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!

On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:

Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/r8k8a

On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

[Yu Li]

Good to know your thoughts and the coming talk in Flink Forward Berlin Oytun, interesting topic and great job! And it's great to hear the voice from application perspective.

Could you share, if possible, the reason why you need to open the state to outside instead of writing the result to sink and directly query there? In another thread there's a case that sink writes to different kafka topics so state is the only place to get a global view, is this the same case you're facing? Or some different requirements? I believe more attention will be drawn to QS if more and more user requirements emerge (smile).

Thanks.

On Wed, 14 Aug 2019 at 00:50, Oytun Tez <[hidden email]> wrote:

Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!

On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:

Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/r8k8a

On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

[Oytun Tez]

Hi Yu,

Excuse my late reply... We simply want Flink to be our centralized stream analysis platform, where we 1) consume data, 2) generate analysis, 3) present analysis. I honestly don't want "stream analysis" to spill out to other components in our ecosystem (e.g., sinking insights into a DB-like place).

So the case for QS for us is centralization of input, output, presentation. State Processor API for instance also counts as a presentation tool for us (on top of migration tool).

This kind of all-in-one (in, out, ui) packaging helps with small teams.

On Wed, Aug 14, 2019 at 3:13 AM Yu Li <[hidden email]> wrote:

Good to know your thoughts and the coming talk in Flink Forward Berlin Oytun, interesting topic and great job! And it's great to hear the voice from application perspective.

Could you share, if possible, the reason why you need to open the state to outside instead of writing the result to sink and directly query there? In another thread there's a case that sink writes to different kafka topics so state is the only place to get a global view, is this the same case you're facing? Or some different requirements? I believe more attention will be drawn to QS if more and more user requirements emerge (smile).

Thanks.

On Wed, 14 Aug 2019 at 00:50, Oytun Tez <[hidden email]> wrote:

Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!

On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:

Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/r8k8a

On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

[Yu Li]

Thanks for the reply Oytun (and sorry for the late response, somehow just noticed).

Requirement received, interesting one. Let's see whether this could draw any attention from the committers (smile).

On Fri, 6 Sep 2019 at 22:14, Oytun Tez <[hidden email]> wrote:

Hi Yu,

Excuse my late reply... We simply want Flink to be our centralized stream analysis platform, where we 1) consume data, 2) generate analysis, 3) present analysis. I honestly don't want "stream analysis" to spill out to other components in our ecosystem (e.g., sinking insights into a DB-like place).

So the case for QS for us is centralization of input, output, presentation. State Processor API for instance also counts as a presentation tool for us (on top of migration tool).

This kind of all-in-one (in, out, ui) packaging helps with small teams.

On Wed, Aug 14, 2019 at 3:13 AM Yu Li <[hidden email]> wrote:

Good to know your thoughts and the coming talk in Flink Forward Berlin Oytun, interesting topic and great job! And it's great to hear the voice from application perspective.

Could you share, if possible, the reason why you need to open the state to outside instead of writing the result to sink and directly query there? In another thread there's a case that sink writes to different kafka topics so state is the only place to get a global view, is this the same case you're facing? Or some different requirements? I believe more attention will be drawn to QS if more and more user requirements emerge (smile).

Thanks.

On Wed, 14 Aug 2019 at 00:50, Oytun Tez <[hidden email]> wrote:

Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!

On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:

Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/r8k8a

On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:

Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).