Re: Queryable State

Posted by Vishal Santoshi on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Queryable-State-tp18973p19001.html

Those are understandable. I am more interested in a few things ( and may be more that could be added ) 

* As far as I can understand JM is the SPOF. Does HA become a necessity ?

* If there are 2 or more JM could we theoretically have a LB fronting them ? Thus it is a peer to peer access ( Cassandra ) or a master slave setup for JM HA specifically for Queryable access ( For  flink jobs it is master slave ) 

* Do we replicate state to other TMs for read optimization ( specifically to avoid Hot Node issues ) ?

* If the job goes down it seems the state is not accessible. What plans to we have to "separate concerns" for Queryable state.

We consider Queryable State significant a feature Flink provides and would do the necessary leg work if there are certain gaps in it being trully considered a Highly Available Key Value store.

Regards.

 

On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <[hidden email]> wrote:

Hi Vishal,

In general, Queryable State should be ready to use.
There are a few things to consider though:

- State queries are not synchronized with the application code, i.e., they can happen at the same time. Therefore, the Flink application should not modify objects that have been put into or read from the state if you are not using the RocksDBStatebackend (which creates copies by deserialization).

- State will be rolled back after a failure. Hence, you can read writes that are not "committed by a checkpoint".

@Kostas, did I forget something?

Best, Fabian

2018-03-18 16:50 GMT+01:00 Vishal Santoshi <[hidden email]>:

To be more precise, is anything thing similar to https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin . done in Samza, can be replicated with production level guarantees with Flink Queryable state ( as it stands currently version 1.5 )  ? 

On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <[hidden email]> wrote:

We are making few decisions on use cases where  Queryable state is a natural fit https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html

Is Queryable state production ready ? We will go to 1.5 flnk if that helps to make the case for the usage.