Running job in "dry mode"?

[Maciek Próchniak]

Hello,

I'd like to be able to see if new version of my job is compatible with
the old one.

I can make a savepoint and run new version from that, but I'd like to be
able to do it without actually starting sources and so on - so that e.g.
it won't start to read from my kafka topics.

Of course I can do it by substituting configuration values, or running
without network access - but this seems a bit mundane and error-prone.

Do you know about any ways to achieve this?

thanks,

maciek

[Tzu-Li (Gordon) Tai]

Hi Maciek,

Is there any particular reason why you do not wish to start running the Kafka sources on the test run?

Otherwise, it would be perfectly fine to start the test job for testing to see if everything works, and keep that savepoint eventually for the non-dry run.

Also, what our integration tests for migrating across Flink versions typically do is have some dummy collection source (`fromElements`) for the test job.

Cheers,

Gordon

On 7 June 2017 at 7:34:25 AM, Maciek Próchniak ([hidden email]) wrote:

Hello,

I'd like to be able to see if new version of my job is compatible with
the old one.

I can make a savepoint and run new version from that, but I'd like to be
able to do it without actually starting sources and so on - so that e.g.
it won't start to read from my kafka topics.

Of course I can do it by substituting configuration values, or running
without network access - but this seems a bit mundane and error-prone.

Do you know about any ways to achieve this?

thanks,

maciek

[Maciek Próchniak]

On 07/06/2017 10:27, Maciek Próchniak wrote:

On 07/06/2017 10:07, Tzu-Li (Gordon) Tai wrote:

Hi Maciek,

Is there any particular reason why you do not wish to start running the Kafka sources on the test run?

Otherwise, it would be perfectly fine to start the test job for testing to see if everything works, and keep that savepoint eventually for the non-dry run.

well, I want to make sure I don't interfere with currently running, production process. While I could use different consumer I certainly don't want to have events emitted. Although it may work if I have some dummy sinks... I'll think about that...

Also, what our integration tests for migrating across Flink versions typically do is have some dummy collection source (`fromElements`) for the test job.

yes, I also tried that. Unfortunately I encountered a problem: when I replace kafka source with collection source the state becomes incompatible, because collection source has different state than kafka one...

thanks,
maciek

Cheers,

Gordon

On 7 June 2017 at 7:34:25 AM, Maciek Próchniak ([hidden email]) wrote:

Hello,

I'd like to be able to see if new version of my job is compatible with
the old one.

I can make a savepoint and run new version from that, but I'd like to be
able to do it without actually starting sources and so on - so that e.g.
it won't start to read from my kafka topics.

Of course I can do it by substituting configuration values, or running
without network access - but this seems a bit mundane and error-prone.

Do you know about any ways to achieve this?

thanks,

maciek