How to schedule Flink Batch Job periodically or daily

Hi Team,

We have Flink Batch Jobs which needs to be scheduled as listed below:

Case 1 :    2.00 UTC time  daily

Case 2 :    Periodically 2 hours once

Case 3:     Schedule based on an event

Request you to help me on this,  How to approach all the 3 use cases. 

Can we use Oozie workflows or any better approach.

Regards,

Sunitha

[rmetzger0]

Hi Sunitha,

(Note: You've emailed both the dev@ and user@ mailing list. Please only use the user@ mailing list for questions on how to use Flink. I'm moving the dev@ list to bcc)

Flink does not have facilities for scheduling batch jobs, and there are no plans to add such a feature (this is not in the scope of Flink, there are already a number of workflow management tools).

On Fri, Sep 11, 2020 at 1:10 PM [hidden email] <[hidden email]> wrote:

Hi Team,

We have Flink Batch Jobs which needs to be scheduled as listed below:

Case 1 :    2.00 UTC time  daily

Case 2 :    Periodically 2 hours once

Case 3:     Schedule based on an event

Request you to help me on this,  How to approach all the 3 use cases. 

Can we use Oozie workflows or any better approach.

Regards,

Sunitha

[Arvid Heise-3]

Hi Sunitha,

oozie is a valid approach, but I'd recommend to evaluate Airflow first [1]. It's much better maintained and easier to use.

Both tools are more used to compose complex workflows though. If you just need a repeated execution, I'd go with cron jobs. For example, you can completely rely on K8s and use cron jobs [2] there if you run on K8s anyways (which I'd recommend).

[1] https://airflow.apache.org/

[2] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

On Fri, Sep 11, 2020 at 1:21 PM Robert Metzger <[hidden email]> wrote:

Hi Sunitha,

(Note: You've emailed both the dev@ and user@ mailing list. Please only use the user@ mailing list for questions on how to use Flink. I'm moving the dev@ list to bcc)

Flink does not have facilities for scheduling batch jobs, and there are no plans to add such a feature (this is not in the scope of Flink, there are already a number of workflow management tools).

On Fri, Sep 11, 2020 at 1:10 PM [hidden email] <[hidden email]> wrote:

Hi Team,

We have Flink Batch Jobs which needs to be scheduled as listed below:

Case 1 :    2.00 UTC time  daily

Case 2 :    Periodically 2 hours once

Case 3:     Schedule based on an event

Request you to help me on this,  How to approach all the 3 use cases. 

Can we use Oozie workflows or any better approach.

Regards,

Sunitha

--

Hi Arvid,

Thanks a lot.

Will check Airflow and Cron-job options.

Regards,

Sunitha.

On Monday, September 14, 2020, 05:23:43 PM GMT+5:30, Arvid Heise <[hidden email]> wrote:

Hi Sunitha,

oozie is a valid approach, but I'd recommend to evaluate Airflow first [1]. It's much better maintained and easier to use.

Both tools are more used to compose complex workflows though. If you just need a repeated execution, I'd go with cron jobs. For example, you can completely rely on K8s and use cron jobs [2] there if you run on K8s anyways (which I'd recommend).

[1] https://airflow.apache.org/

[2] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

On Fri, Sep 11, 2020 at 1:21 PM Robert Metzger <[hidden email]> wrote:

Hi Sunitha,

(Note: You've emailed both the dev@ and user@ mailing list. Please only use the user@ mailing list for questions on how to use Flink. I'm moving the dev@ list to bcc)

Flink does not have facilities for scheduling batch jobs, and there are no plans to add such a feature (this is not in the scope of Flink, there are already a number of workflow management tools).

On Fri, Sep 11, 2020 at 1:10 PM [hidden email] <[hidden email]> wrote:

Hi Team,

We have Flink Batch Jobs which needs to be scheduled as listed below:

Case 1 :    2.00 UTC time  daily

Case 2 :    Periodically 2 hours once

Case 3:     Schedule based on an event

Request you to help me on this,  How to approach all the 3 use cases. 

Can we use Oozie workflows or any better approach.

Regards,

Sunitha

--