Implicit Flink Context Documentation

[Padarn Wilson-2]

Hi Users,

I am trying to understand the details of how some aspects of Flink work.

While understanding `keyed state` I kept coming up against a claim that `there is a specific key implicitly in context` I would like to understand how this works, which I'm guessing means understanding the details of the runtime context: Is there any documentation or FLIP someone can recommend on this?  

[Piotr Nowojski-3]

Hi,

Please take a look for example here:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state

And the example in particular

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state

The part about "there is a specific key implicitly in context” might be referring to the fact, that for every instance of `CountWindowAverage` that will be running in the cluster, user doesn’t have to set the key context explicility. Flink will set the the key context automatically for the `ValueState<Tuple2<Long, Long>> sum;` before any invocation of `CountWindowAverage#flatMap` method.

In other words, one parallel instance of `CountWindowAverage` function, for two consecutive invocations of `CountWindowAverage#flatMap` can be referring to different underlying value of `CountWindowAverage#sum` field. For details you could take a look at `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` method and how it’s being used/implemented.

I hope that helps.

Piotrek

On 13 Mar 2020, at 08:20, Padarn Wilson <[hidden email]> wrote:

Hi Users,

I am trying to understand the details of how some aspects of Flink work.

While understanding `keyed state` I kept coming up against a claim that `there is a specific key implicitly in context` I would like to understand how this works, which I'm guessing means understanding the details of the runtime context: Is there any documentation or FLIP someone can recommend on this?  

[Padarn Wilson-2]

Thanks Piotr,

Conceptually I understand (and use) the key'ed state quite a lot, but the implementation details are what I was looking for.

It looks like `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` is what I'm looking for though. It would be cool if there were some internals design doc however? Quite hard to dig through the code as there is a log tied to how the execution of the job actually happens.

Padarn

On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski <[hidden email]> wrote:

Hi,

Please take a look for example here:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state

And the example in particular

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state

The part about "there is a specific key implicitly in context” might be referring to the fact, that for every instance of `CountWindowAverage` that will be running in the cluster, user doesn’t have to set the key context explicility. Flink will set the the key context automatically for the `ValueState<Tuple2<Long, Long>> sum;` before any invocation of `CountWindowAverage#flatMap` method.

In other words, one parallel instance of `CountWindowAverage` function, for two consecutive invocations of `CountWindowAverage#flatMap` can be referring to different underlying value of `CountWindowAverage#sum` field. For details you could take a look at `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` method and how it’s being used/implemented.

I hope that helps.

Piotrek

On 13 Mar 2020, at 08:20, Padarn Wilson <[hidden email]> wrote:

Hi Users,

I am trying to understand the details of how some aspects of Flink work.

While understanding `keyed state` I kept coming up against a claim that `there is a specific key implicitly in context` I would like to understand how this works, which I'm guessing means understanding the details of the runtime context: Is there any documentation or FLIP someone can recommend on this?  

[Piotr Nowojski-3]

Hi,

We are not maintaining internal docs. We have design docs for newly proposed features (previously informal design docs published on dev mailing list and recently as FLIP documents [1]), but keyed state is such an old concept that dates back so much into the past, that I’m pretty sure it pre dates any of that. So you would have to digg through the code if you want to understand it.

Piotrek

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On 13 Mar 2020, at 16:14, Padarn Wilson <[hidden email]> wrote:

Thanks Piotr,

Conceptually I understand (and use) the key'ed state quite a lot, but the implementation details are what I was looking for.

It looks like `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` is what I'm looking for though. It would be cool if there were some internals design doc however? Quite hard to dig through the code as there is a log tied to how the execution of the job actually happens.

Padarn

On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski <[hidden email]> wrote:

Hi,

Please take a look for example here:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state

And the example in particular

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state

The part about "there is a specific key implicitly in context” might be referring to the fact, that for every instance of `CountWindowAverage` that will be running in the cluster, user doesn’t have to set the key context explicility. Flink will set the the key context automatically for the `ValueState<Tuple2<Long, Long>> sum;` before any invocation of `CountWindowAverage#flatMap` method.

In other words, one parallel instance of `CountWindowAverage` function, for two consecutive invocations of `CountWindowAverage#flatMap` can be referring to different underlying value of `CountWindowAverage#sum` field. For details you could take a look at `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` method and how it’s being used/implemented.

I hope that helps.

Piotrek

On 13 Mar 2020, at 08:20, Padarn Wilson <[hidden email]> wrote:

Hi Users,

I am trying to understand the details of how some aspects of Flink work.

While understanding `keyed state` I kept coming up against a claim that `there is a specific key implicitly in context` I would like to understand how this works, which I'm guessing means understanding the details of the runtime context: Is there any documentation or FLIP someone can recommend on this?  

[Padarn Wilson-2]

Thanks for the clarification. I'll dig in then!

On Mon, 16 Mar 2020, 3:47 pm Piotr Nowojski, <[hidden email]> wrote:

Hi,

We are not maintaining internal docs. We have design docs for newly proposed features (previously informal design docs published on dev mailing list and recently as FLIP documents [1]), but keyed state is such an old concept that dates back so much into the past, that I’m pretty sure it pre dates any of that. So you would have to digg through the code if you want to understand it.

Piotrek

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On 13 Mar 2020, at 16:14, Padarn Wilson <[hidden email]> wrote:

Thanks Piotr,

Conceptually I understand (and use) the key'ed state quite a lot, but the implementation details are what I was looking for.

It looks like `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` is what I'm looking for though. It would be cool if there were some internals design doc however? Quite hard to dig through the code as there is a log tied to how the execution of the job actually happens.

Padarn

On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski <[hidden email]> wrote:

Hi,

Please take a look for example here:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state

And the example in particular

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state

The part about "there is a specific key implicitly in context” might be referring to the fact, that for every instance of `CountWindowAverage` that will be running in the cluster, user doesn’t have to set the key context explicility. Flink will set the the key context automatically for the `ValueState<Tuple2<Long, Long>> sum;` before any invocation of `CountWindowAverage#flatMap` method.

In other words, one parallel instance of `CountWindowAverage` function, for two consecutive invocations of `CountWindowAverage#flatMap` can be referring to different underlying value of `CountWindowAverage#sum` field. For details you could take a look at `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1` method and how it’s being used/implemented.

I hope that helps.

Piotrek

On 13 Mar 2020, at 08:20, Padarn Wilson <[hidden email]> wrote:

Hi Users,

I am trying to understand the details of how some aspects of Flink work.

While understanding `keyed state` I kept coming up against a claim that `there is a specific key implicitly in context` I would like to understand how this works, which I'm guessing means understanding the details of the runtime context: Is there any documentation or FLIP someone can recommend on this?