(DEPRECATED) Apache Flink User Mailing List archive.

Programmatic management of Flink jobs

classic Classic list List threaded Threaded
♦
♦
 9 messages Options
Moiz Jinia
Reply | Threaded
Open this post in threaded view
♦
♦
|

Programmatic management of Flink jobs

Moiz Jinia
Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Aljoscha Krettek
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Aljoscha Krettek
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Moiz Jinia
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Moiz Jinia
Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Aljoscha Krettek
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Aljoscha Krettek
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 
On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Moiz Jinia
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Moiz Jinia
Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <[hidden email]> wrote:
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 

On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Moiz Jinia
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Moiz Jinia
The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.

Does that sound like it'd work or am I missing something?

Moiz

On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <[hidden email]> wrote:
Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <[hidden email]> wrote:
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 

On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Aljoscha Krettek
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Aljoscha Krettek
Hi,
Yes, this would work even though it requires canceling/restarting the job whenever the patterns change.

Best,
Aljoscha
On 3. May 2017, at 12:36, Moiz S Jinia <[hidden email]> wrote:

The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.

Does that sound like it'd work or am I missing something?

Moiz

On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <[hidden email]> wrote:
Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <[hidden email]> wrote:
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 

On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Moiz Jinia
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Moiz Jinia
Thanks. Do you foresee any problems with canceling/starting the jobs whenever the patterns change?

Moiz

On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
Yes, this would work even though it requires canceling/restarting the job whenever the patterns change.

Best,
Aljoscha

On 3. May 2017, at 12:36, Moiz S Jinia <[hidden email]> wrote:

The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.

Does that sound like it'd work or am I missing something?

Moiz

On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <[hidden email]> wrote:
Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <[hidden email]> wrote:
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 

On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Aljoscha Krettek
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Programmatic management of Flink jobs

Aljoscha Krettek
Hi,
Not if it doesn’t happen to frequently. Otherwise you would spend most of your time restarting jobs.

Best,
Aljoscha

On 9. May 2017, at 08:23, Moiz S Jinia <[hidden email]> wrote:

Thanks. Do you foresee any problems with canceling/starting the jobs whenever the patterns change?

Moiz

On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
Yes, this would work even though it requires canceling/restarting the job whenever the patterns change.

Best,
Aljoscha

On 3. May 2017, at 12:36, Moiz S Jinia <[hidden email]> wrote:

The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.

Does that sound like it'd work or am I missing something?

Moiz

On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <[hidden email]> wrote:
Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <[hidden email]> wrote:
What would the pattern be added to. An existing custom operator?

The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure. 

On 3. May 2017, at 11:43, Moiz S Jinia <[hidden email]> wrote:

Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?

On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.


On 3. May 2017, at 10:02, Moiz S Jinia <[hidden email]> wrote:

Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.


Moiz
Free forum by Nabble Edit this page