ListState<T> - elements order

[Alexey Trenikhun]

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[vino yang]

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Aljoscha Krettek]

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[vino yang]

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Rong Rong]

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge.

I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default.

--

Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[vino yang]

Hi,

I saw one of ListState's implementations of HeapListState, and its internal data store uses the JDK's List. 

Of course, from an API point of view, maybe we can't make an absolute order guarantee. 

But if we look at it from a particular implementation, we can see if it can guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong <[hidden email]> 于2018年9月14日周五 下午12:30写道：

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge.

I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default.

--

Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Vijay Bhaskar]

How it would be to use ValueState<T> with values as string separated by the delimiter. So that order will never be a problem. Only overhead is to separate delimiter, read the elements and convert them into primitive types in case necessary. It just workaround. In case doesn't suite your requirements pardon me

Regards

Bhaskar

On Fri, Sep 14, 2018 at 12:36 PM vino yang <[hidden email]> wrote:

Hi,

I saw one of ListState's implementations of HeapListState, and its internal data store uses the JDK's List. 

Of course, from an API point of view, maybe we can't make an absolute order guarantee. 

But if we look at it from a particular implementation, we can see if it can guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong <[hidden email]> 于2018年9月14日周五 下午12:30写道：

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge.

I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default.

--

Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[vino yang]

Yes, for strings, we can do this, but it is not generic enough. But your idea reminds me that we decide which data structure to use, if we use ValueState<T>.

Vijay Bhaskar <[hidden email]> 于2018年9月14日周五 下午5:24写道：

How it would be to use ValueState<T> with values as string separated by the delimiter. So that order will never be a problem. Only overhead is to separate delimiter, read the elements and convert them into primitive types in case necessary. It just workaround. In case doesn't suite your requirements pardon me

Regards

Bhaskar

On Fri, Sep 14, 2018 at 12:36 PM vino yang <[hidden email]> wrote:

Hi,

I saw one of ListState's implementations of HeapListState, and its internal data store uses the JDK's List. 

Of course, from an API point of view, maybe we can't make an absolute order guarantee. 

But if we look at it from a particular implementation, we can see if it can guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong <[hidden email]> 于2018年9月14日周五 下午12:30写道：

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge.

I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default.

--

Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Alexey Trenikhun]

“ValueState<T> with values as string separated by the delimiter” - is not necessary, it can be ValueState<List<T>> instead. Drawback of using ValueState that it is necessary to serialize whole contained object when at least one element is added.

Alexey

 

---

From: Vijay Bhaskar <[hidden email]>
Sent: Friday, September 14, 2018 2:24 AM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: Re: ListState - elements order

 

How it would be to use ValueState<T> with values as string separated by the delimiter. So that order will never be a problem. Only overhead is to separate delimiter, read the elements and convert them into primitive types in case necessary. It just workaround. In case doesn't suite your requirements pardon me

Regards

Bhaskar

On Fri, Sep 14, 2018 at 12:36 PM vino yang <[hidden email]> wrote:

Hi,

I saw one of ListState's implementations of HeapListState, and its internal data store uses the JDK's List. 

Of course, from an API point of view, maybe we can't make an absolute order guarantee. 

But if we look at it from a particular implementation, we can see if it can guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong <[hidden email]> 于2018年9月14日周五 下午12:30写道：

I don't think ordering is guaranteed in the internal implementation, to the best of my knowledge.

I agreed with Aljoscha, if there is no clear definition of ordering, it is assumed to be not preserved by default.

--

Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Kostas Kloudas]

Hi all,

Flink does not provide any guarantees about the order of the elements in a list and it leaves it to the state-backends.

This means that semantics between different backends may differ, and even if something holds now for one of them, if 

RocksDB or a filesystem decides to change its semantics, then any assumptions will not hold anymore.

So I would recommend to program under the assumption that no order guarantees are provided.

Cheers,

Kostas

On Sep 14, 2018, at 4:30 AM, vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey

[Kostas Kloudas]

Oops,

Sorry but I lost part of the discussion that had already been made.

Please ignore my previous answer.

Kostas

On Sep 17, 2018, at 4:37 PM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Flink does not provide any guarantees about the order of the elements in a list and it leaves it to the state-backends.

This means that semantics between different backends may differ, and even if something holds now for one of them, if 

RocksDB or a filesystem decides to change its semantics, then any assumptions will not hold anymore.

So I would recommend to program under the assumption that no order guarantees are provided.

Cheers,

Kostas

On Sep 14, 2018, at 4:30 AM, vino yang <[hidden email]> wrote:

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some internal work. 

But if it's just for the user, isn't it clear without any internal operations? I think if the user explicitly uses it, it should conform to the basic List semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek <[hidden email]> 于2018年9月13日周四 下午10:42写道：

Hi,

this is not clearly defined anywhere, and I was always working under the assumption that the order is not preserved. This potentially allows more optimizations by the system, and for example in case of merging windows we don't know the order of elements in a ListState after a merge.

Best,

Aljoscha

On 6. Sep 2018, at 08:19, vino yang <[hidden email]> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of elements.

Thank, vino.

Alexey Trenikhun <[hidden email]> 于2018年9月6日周四 上午10:55写道：

Hello,

Does keyed managed ListState<T> preserve elements order, for example if I call listState.add(e1); listState.add(e2); listState.add(e3); , does ListState<T> guarantee that listState.get() will return elements in order they were added (e1, e2, e3)

Alexey