Queryable state and TTL

[avilevi]

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi

[Andrey Zagrebin-3]

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to work on it for the next releases.

I think the community will firstly work on the event time support for TTL.

I will loop Yu in, maybe he has some plans to work on TTL for the queryable state.

Best,

Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi <[hidden email]> wrote:

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi

[Yu Li-2]

Thanks for the ping Andrey.

For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1].

Although we could see emerging requirements and proposals on QueryableState recently, prioritizing is important for each open source project. And personally I think it may help if we could gather more and clearly describe the other-than-debugging use cases of QueryableState in production [2]. Could you share your case with us and why QueryableState is necessary rather than querying the data from sink @Avi? Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/hJDA

On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin <[hidden email]> wrote:

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to work on it for the next releases.

I think the community will firstly work on the event time support for TTL.

I will loop Yu in, maybe he has some plans to work on TTL for the queryable state.

Best,

Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi <[hidden email]> wrote:

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi

[avilevi]

Hi Yu,

Our sink is actually Kafka hence we cannot query it properly, from there we distribute it to different consumers. We keep info in our state such as entry time, some accumulated data etc' , this data is not kept elsewhere hence we need to query our state. 

Best regards

Avi

On Thu, Jul 4, 2019 at 7:20 AM Yu Li <[hidden email]> wrote:

This Message originated outside your organization.

---

Thanks for the ping Andrey.

For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1].

Although we could see emerging requirements and proposals on QueryableState recently, prioritizing is important for each open source project. And personally I think it may help if we could gather more and clearly describe the other-than-debugging use cases of QueryableState in production [2]. Could you share your case with us and why QueryableState is necessary rather than querying the data from sink @Avi? Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/hJDA

On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin <[hidden email]> wrote:

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to work on it for the next releases.

I think the community will firstly work on the event time support for TTL.

I will loop Yu in, maybe he has some plans to work on TTL for the queryable state.

Best,

Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi <[hidden email]> wrote:

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi

[Eron Wright]

Here's a PR for queryable state TLS that I closed because I didn't have time, and because I get the impression that the queryable state feature is used very often.    Feel free to take it up, if you like.

https://github.com/apache/flink/pull/6626

-Eron

On Wed, Jul 3, 2019 at 11:21 PM Avi Levi <[hidden email]> wrote:

Hi Yu,

Our sink is actually Kafka hence we cannot query it properly, from there we distribute it to different consumers. We keep info in our state such as entry time, some accumulated data etc' , this data is not kept elsewhere hence we need to query our state. 

Best regards

Avi

On Thu, Jul 4, 2019 at 7:20 AM Yu Li <[hidden email]> wrote:

This Message originated outside your organization.

---

Thanks for the ping Andrey.

For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1].

Although we could see emerging requirements and proposals on QueryableState recently, prioritizing is important for each open source project. And personally I think it may help if we could gather more and clearly describe the other-than-debugging use cases of QueryableState in production [2]. Could you share your case with us and why QueryableState is necessary rather than querying the data from sink @Avi? Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/hJDA

On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin <[hidden email]> wrote:

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to work on it for the next releases.

I think the community will firstly work on the event time support for TTL.

I will loop Yu in, maybe he has some plans to work on TTL for the queryable state.

Best,

Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi <[hidden email]> wrote:

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi

[avilevi]

Thanks, I'll check it out. 

On Sun, Jul 7, 2019 at 5:40 AM Eron Wright <[hidden email]> wrote:

This Message originated outside your organization.

---

Here's a PR for queryable state TLS that I closed because I didn't have time, and because I get the impression that the queryable state feature is used very often.    Feel free to take it up, if you like.

https://github.com/apache/flink/pull/6626

-Eron

On Wed, Jul 3, 2019 at 11:21 PM Avi Levi <[hidden email]> wrote:

Hi Yu,

Our sink is actually Kafka hence we cannot query it properly, from there we distribute it to different consumers. We keep info in our state such as entry time, some accumulated data etc' , this data is not kept elsewhere hence we need to query our state. 

Best regards

Avi

On Thu, Jul 4, 2019 at 7:20 AM Yu Li <[hidden email]> wrote:

This Message originated outside your organization.

---

Thanks for the ping Andrey.

For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1].

Although we could see emerging requirements and proposals on QueryableState recently, prioritizing is important for each open source project. And personally I think it may help if we could gather more and clearly describe the other-than-debugging use cases of QueryableState in production [2]. Could you share your case with us and why QueryableState is necessary rather than querying the data from sink @Avi? Thanks.

[1] https://s.apache.org/MaOl

[2] https://s.apache.org/hJDA

On Wed, 3 Jul 2019 at 23:13, Andrey Zagrebin <[hidden email]> wrote:

Hi Avi,

It is on the road map but I am not aware about plans of any contributor to work on it for the next releases.

I think the community will firstly work on the event time support for TTL.

I will loop Yu in, maybe he has some plans to work on TTL for the queryable state.

Best,

Andrey

On Wed, Jul 3, 2019 at 3:17 PM Avi Levi <[hidden email]> wrote:

Hi,

Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL)

I saw in previous mailing thread that it is on the roadmap. Is it still on the roadmap ?

* There is a workaround which is using timers to clear the state, but in our case, it means firing billons of timers on daily basis all at the same time, which seems no to very efficient and might cause some resources issues  

Cheers

Avi