Migrating existing application to Flink

[Eduardo Winpenny Tejedor]

Hi all,

How would one go about migrating a stateful streaming application that doesn't use Flink to one that does?

My question is specifically how to load state for the first time? I can set the source operator (Kafka in my case) to start from a desired point in time but I've no idea how I'd go about initialising the state of the Flink equivalent version of the app from a "pseudo-snapshot". I call "pseudo-snapshot" to the non-Flink representation of state at a given point in time.

Thanks,

Eduardo

[Konstantin Knauf-2]

Hi Eduardo,

Flink 1.9 will add a new State Processor API [1], which you can use to create Savepoints from scratch with a batch job. 

Cheers,

Konstantin

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/state_processor_api.html#writing-new-savepoints

On Thu, Jul 4, 2019 at 12:38 AM Eduardo Winpenny Tejedor <[hidden email]> wrote:

Hi all,

How would one go about migrating a stateful streaming application that doesn't use Flink to one that does?

My question is specifically how to load state for the first time? I can set the source operator (Kafka in my case) to start from a desired point in time but I've no idea how I'd go about initialising the state of the Flink equivalent version of the app from a "pseudo-snapshot". I call "pseudo-snapshot" to the non-Flink representation of state at a given point in time.

Thanks,

Eduardo

--

[Eduardo Winpenny Tejedor]

Great news! Thanks Konstantin

On Sun, 7 Jul 2019, 09:34 Konstantin Knauf, <[hidden email]> wrote:

Hi Eduardo,

Flink 1.9 will add a new State Processor API [1], which you can use to create Savepoints from scratch with a batch job. 

Cheers,

Konstantin

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/state_processor_api.html#writing-new-savepoints

On Thu, Jul 4, 2019 at 12:38 AM Eduardo Winpenny Tejedor <[hidden email]> wrote:

Hi all,

How would one go about migrating a stateful streaming application that doesn't use Flink to one that does?

My question is specifically how to load state for the first time? I can set the source operator (Kafka in my case) to start from a desired point in time but I've no idea how I'd go about initialising the state of the Flink equivalent version of the app from a "pseudo-snapshot". I call "pseudo-snapshot" to the non-Flink representation of state at a given point in time.

Thanks,

Eduardo

--

[Oytun Tez]

I am so excited with 1.9, State Processor API is single handedly the most important update for me.

On Sun, Jul 7, 2019 at 4:34 AM Konstantin Knauf <[hidden email]> wrote:

Hi Eduardo,

Flink 1.9 will add a new State Processor API [1], which you can use to create Savepoints from scratch with a batch job. 

Cheers,

Konstantin

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/state_processor_api.html#writing-new-savepoints

On Thu, Jul 4, 2019 at 12:38 AM Eduardo Winpenny Tejedor <[hidden email]> wrote:

Hi all,

How would one go about migrating a stateful streaming application that doesn't use Flink to one that does?

My question is specifically how to load state for the first time? I can set the source operator (Kafka in my case) to start from a desired point in time but I've no idea how I'd go about initialising the state of the Flink equivalent version of the app from a "pseudo-snapshot". I call "pseudo-snapshot" to the non-Flink representation of state at a given point in time.

Thanks,

Eduardo

--