Managing Jobs entirely with Flink Monitoring API

[Barak Ben Nathan]

 

I want to manage the execution of Flink Jobs programmatically through Flink Monitoring API.

 

I.e. I want to run/delete jobs ONLY with the
 POST /jars/:jarid/run  
 POST /jobs/:jobid/stop
API commands.

 

Now, it seems that the Session Mode may fits my needs: “Session Mode: one JobManager instance manages multiple jobs sharing the same cluster of TaskManagers” (https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/)

However, I couldn’t find a way to start the API server (i.e. a JobManager) that didn’t already include submitting a JAR file for a job execution.

Any suggestions?

[Piotr Nowojski-5]

Hi Barak,

Before starting the JobManager I don't think there is any API running at all. If you want to be able to submit/stop multiple jobs to the same cluster session mode is indeed the way to go. But first you need to to start the cluster ( start-cluster.sh ) [1]

Piotrek

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/

wt., 25 maj 2021 o 14:10 Barak Ben Nathan <[hidden email]> napisał(a):

 

I want to manage the execution of Flink Jobs programmatically through Flink Monitoring API.

 

I.e. I want to run/delete jobs ONLY with the
 POST /jars/:jarid/run  
 POST /jobs/:jobid/stop
API commands.

 

Now, it seems that the Session Mode may fits my needs: “Session Mode: one JobManager instance manages multiple jobs sharing the same cluster of TaskManagers” (https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/)

However, I couldn’t find a way to start the API server (i.e. a JobManager) that didn’t already include submitting a JAR file for a job execution.

Any suggestions?

[Piotr Nowojski-4]

Glad to hear it!  Thanks for confirming that it works.

Piotrek

śr., 26 maj 2021 o 12:59 Barak Ben Nathan <[hidden email]> napisał(a):

 

Hi Piotrek,

 

This is exactly what I was searching for. Thanks!

 

Barak

 

From: Piotr Nowojski <[hidden email]>
Sent: Wednesday, May 26, 2021 9:59 AM
To: Barak Ben Nathan <[hidden email]>
Cc: [hidden email]
Subject: Re: Managing Jobs entirely with Flink Monitoring API

 

CAUTION: external source

Hi Barak,

 

Before starting the JobManager I don't think there is any API running at all. If you want to be able to submit/stop multiple jobs to the same cluster session mode is indeed the way to go. But first you need to to start the cluster ( start-cluster.sh ) [1]

 

Piotrek

 

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/

 

wt., 25 maj 2021 o 14:10 Barak Ben Nathan <[hidden email]> napisał(a):

 

I want to manage the execution of Flink Jobs programmatically through Flink Monitoring API.

 

I.e. I want to run/delete jobs ONLY with the
 POST /jars/:jarid/run  
 POST /jobs/:jobid/stop
API commands.

 

Now, it seems that the Session Mode may fits my needs: “Session Mode: one JobManager instance manages multiple jobs sharing the same cluster of TaskManagers” (https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/)

However, I couldn’t find a way to start the API server (i.e. a JobManager) that didn’t already include submitting a JAR file for a job execution.

Any suggestions?

Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious.