Custom scheduler in Flink

[Felipe Gutierrez]

Hi,

I want to develop a custom scheduler in Flink to be aware for which host Flink must process some task. This post shows (using Apache Storm) the kind of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e).

I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html) and I wonder if there is something more actual or better explained about a custom scheduler.

Thanks,

Felipe

[Chesnay Schepler]

There's no reasonable way to implement a custom Scheduler, i.e., something where can just plug in your scheduler in a nice way. For this you'll have to directly modify the source of Flink.

The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be of interest, but is still in the design phase.

On 28.11.2018 10:21, Felipe Gutierrez wrote:

Hi,

I want to develop a custom scheduler in Flink to be aware for which host
Flink must process some task. This post shows (using Apache Storm) the kind
of example I want to build (
https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
).

I saw some related question on this post from 2016 (
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
and I wonder if there is something more actual or better explained about a
custom scheduler.

Thanks,
Felipe
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*

[Felipe Gutierrez]

Thanks, I'll check it out. 

On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <[hidden email]> wrote:

There's no reasonable way to implement a custom Scheduler, i.e., something where can just plug in your scheduler in a nice way. For this you'll have to directly modify the source of Flink.

The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be of interest, but is still in the design phase.

On 28.11.2018 10:21, Felipe Gutierrez wrote:

Hi,

I want to develop a custom scheduler in Flink to be aware for which host
Flink must process some task. This post shows (using Apache Storm) the kind
of example I want to build (
https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
).

I saw some related question on this post from 2016 (
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
and I wonder if there is something more actual or better explained about a
custom scheduler.

Thanks,
Felipe
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*

[Till Rohrmann]

Hi Felipe,

https://issues.apache.org/jira/browse/FLINK-10429 might also be interesting. The community is currently working on making the Scheduler pluggable to make it easier to extend this component.

Cheers,

Till

On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <[hidden email]> wrote:

Thanks, I'll check it out.
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*


On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <[hidden email]> wrote:

> There's no *reasonable *way to implement a custom Scheduler, i.e.,
> something where can just plug in your scheduler in a nice way. For this
> you'll have to directly modify the source of Flink.
>
> The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> of interest, but is still in the design phase.
>
> On 28.11.2018 10:21, Felipe Gutierrez wrote:
>
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> <https://felipeogutierrez.blogspot.com>*
>
>
>
>

[Felipe Gutierrez]

thanks. I will check it out

On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <[hidden email]> wrote:

Hi Felipe,

https://issues.apache.org/jira/browse/FLINK-10429 might also be interesting. The community is currently working on making the Scheduler pluggable to make it easier to extend this component.

Cheers,

Till

On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <[hidden email]> wrote:

Thanks, I'll check it out.
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*


On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <[hidden email]> wrote:

> There's no *reasonable *way to implement a custom Scheduler, i.e.,
> something where can just plug in your scheduler in a nice way. For this
> you'll have to directly modify the source of Flink.
>
> The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> of interest, but is still in the design phase.
>
> On 28.11.2018 10:21, Felipe Gutierrez wrote:
>
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> <https://felipeogutierrez.blogspot.com>*
>
>
>
>

[Felipe Gutierrez]

thanks. I saw the google docs just now. I am gonna print and study it at the weekend.

On Fri, Nov 30, 2018 at 4:42 PM Felipe Gutierrez <[hidden email]> wrote:

thanks. I will check it out

On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <[hidden email]> wrote:

Hi Felipe,

https://issues.apache.org/jira/browse/FLINK-10429 might also be interesting. The community is currently working on making the Scheduler pluggable to make it easier to extend this component.

Cheers,

Till

On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <[hidden email]> wrote:

Thanks, I'll check it out.
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*


On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <[hidden email]> wrote:

> There's no *reasonable *way to implement a custom Scheduler, i.e.,
> something where can just plug in your scheduler in a nice way. For this
> you'll have to directly modify the source of Flink.
>
> The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> of interest, but is still in the design phase.
>
> On 28.11.2018 10:21, Felipe Gutierrez wrote:
>
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> <https://felipeogutierrez.blogspot.com>*
>
>
>
>