Production readiness of Flink Job Stop Service

[chiggi_dev]

Hi,

I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint. 

I know that since Flink 1.4.2, Stop is not stable/not production ready. 

With Flink 1.5 can it be used for stopping jobs?

Regards,

Chirag 

[vino yang]

Hi Chirag,

Did you read the latest stable Flink documentation about Savepoint[1] and Cancel with savepoint[2] and Upgrade application[3]?

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/savepoints.html#resuming-from-savepoints

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/rest_api.html#cancel-job-with-savepoint

[3]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/upgrading.html

Thanks, vino.

2018-07-19 14:25 GMT+08:00 Chirag Dewan <[hidden email]>:

Hi,

I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint. 

I know that since Flink 1.4.2, Stop is not stable/not production ready. 

With Flink 1.5 can it be used for stopping jobs?

Regards,

Chirag 

[Fabian Hueske-2]

Hi Chirag,

Stop with savepoint is not mentioned in the 1.5.0 release notes [1].

Since its a frequently requested feature, I'm pretty sure that it would have been mentioned if it was added.

Best, Fabian

[1] http://flink.apache.org/news/2018/05/25/release-1.5.0.html

2018-07-19 8:39 GMT+02:00 vino yang <[hidden email]>:

Hi Chirag,

Did you read the latest stable Flink documentation about Savepoint[1] and Cancel with savepoint[2] and Upgrade application[3]?

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/savepoints.html#resuming-from-savepoints

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/rest_api.html#cancel-job-with-savepoint

[3]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/upgrading.html

Thanks, vino.

2018-07-19 14:25 GMT+08:00 Chirag Dewan <[hidden email]>:

Hi,

I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint. 

I know that since Flink 1.4.2, Stop is not stable/not production ready. 

With Flink 1.5 can it be used for stopping jobs?

Regards,

Chirag 

[chiggi_dev]

Thanks a lot Fabian and Vino.

Is there anyway I can do that without stop? Although the plan should be to use a Two Phase commit sink so that there are no duplicates, I am looking for stopping the sources before the savepoint is triggered.

Thanks,

Chirag

Sent from Yahoo Mail on Android

On Thu, 19 Jul 2018 at 1:35 PM, Fabian Hueske

<[hidden email]> wrote:

Hi Chirag,

Stop with savepoint is not mentioned in the 1.5.0 release notes [1].

Since its a frequently requested feature, I'm pretty sure that it would have been mentioned if it was added.

Best, Fabian

[1] http://flink.apache.org/news/2018/05/25/release-1.5.0.html

2018-07-19 8:39 GMT+02:00 vino yang <[hidden email]>:

Hi Chirag,

Did you read the latest stable Flink documentation about Savepoint[1] and Cancel with savepoint[2] and Upgrade application[3]?

[1]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/ops/state/ savepoints.html#resuming-from- savepoints

[2]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/monitoring/rest_ api.html#cancel-job-with- savepoint

[3]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/ops/upgrading.html

Thanks, vino.

2018-07-19 14:25 GMT+08:00 Chirag Dewan <[hidden email]>:

Hi,

I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint. 

I know that since Flink 1.4.2, Stop is not stable/not production ready. 

With Flink 1.5 can it be used for stopping jobs?

Regards,

Chirag 

[vino yang]

Hi Chirag,

Actually, if you want to use "stop/resume/pause" pattern, you can trigger a savepoint before stopping job, then you can resume your job with specify this savepoint. The savepoint does not bind with cancel command, although you can only invoke "cancel with savepoint" now.

Now you mentioned "how to implement a sink to make it no duplicate records" and "stop source before the savepoint is triggered", not only about "stop with savepoint". Can you describe your use case more clear or provide more detail information?

Here is a mailing list about discussing "cancel and stop"[1].

[1]: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-flick-cancel-vs-stop-td15534.html

Thanks, vino. 

2018-07-23 10:38 GMT+08:00 Chirag Dewan <[hidden email]>:

Thanks a lot Fabian and Vino.

Is there anyway I can do that without stop? Although the plan should be to use a Two Phase commit sink so that there are no duplicates, I am looking for stopping the sources before the savepoint is triggered.

Thanks,

Chirag

Sent from Yahoo Mail on Android

On Thu, 19 Jul 2018 at 1:35 PM, Fabian Hueske

<[hidden email]> wrote:

Hi Chirag,

Stop with savepoint is not mentioned in the 1.5.0 release notes [1].

Since its a frequently requested feature, I'm pretty sure that it would have been mentioned if it was added.

Best, Fabian

[1] http://flink.apache.org/news/2018/05/25/release-1.5.0.html

2018-07-19 8:39 GMT+02:00 vino yang <[hidden email]>:

Hi Chirag,

Did you read the latest stable Flink documentation about Savepoint[1] and Cancel with savepoint[2] and Upgrade application[3]?

[1]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/ops/state/ savepoints.html#resuming-from- savepoints

[2]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/monitoring/rest_ api.html#cancel-job-with- savepoint

[3]: https://ci.apache.org/ projects/flink/flink-docs- release-1.5/ops/upgrading.html

Thanks, vino.

2018-07-19 14:25 GMT+08:00 Chirag Dewan <[hidden email]>:

Hi,

I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint. 

I know that since Flink 1.4.2, Stop is not stable/not production ready. 

With Flink 1.5 can it be used for stopping jobs?

Regards,

Chirag