Enable custom REST APIs in Flink

[Flavio Pompermaier]

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

[Oytun Tez]

I would LOVE this. We had to hack our way a lot to achieve something similar.

@Flavio, we basically added a new entrypoint to the same codebase and ran that separately in its own container.

On Tue, Apr 21, 2020 at 8:30 AM Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

[Jeff Zhang]

Hi Flavio,

I am curious know where service run, Do you create this service in UDF and run it  in TM ?

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午8:30写道：

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

--

[Flavio Pompermaier]

In my mind the user API could run everywhere but the simplest thing is to make them available in the Job Manager (where the other REST API lives).

They could become a very simple but powerful way to add valuable services to Flink without adding useless complexity to the overall architecture for just a few methods.

I don't know whether Spark or Beam allow you to do something like that but IMHO it's super useful (especially from a maintenance point of view wrt the overall architecture complexity).

@Oytun indeed we'd like to avoid recompiling everything when a single user class (i.e. not related to Flink classes) is modified or added. Glad to see that there are other people having the same problem here

On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <[hidden email]> wrote:

Hi Flavio,

I am curious know where service run, Do you create this service in UDF and run it  in TM ?

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午8:30写道：

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

--

Best Regards

Jeff Zhang

[Oytun Tez]

definitely, this is for me about making Flink an "application framework" rather than solely a "dataflow framework".

On Tue, Apr 21, 2020 at 11:07 AM Flavio Pompermaier <[hidden email]> wrote:

In my mind the user API could run everywhere but the simplest thing is to make them available in the Job Manager (where the other REST API lives).

They could become a very simple but powerful way to add valuable services to Flink without adding useless complexity to the overall architecture for just a few methods.

I don't know whether Spark or Beam allow you to do something like that but IMHO it's super useful (especially from a maintenance point of view wrt the overall architecture complexity).

@Oytun indeed we'd like to avoid recompiling everything when a single user class (i.e. not related to Flink classes) is modified or added. Glad to see that there are other people having the same problem here

On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <[hidden email]> wrote:

Hi Flavio,

I am curious know where service run, Do you create this service in UDF and run it  in TM ?

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午8:30写道：

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

--

Best Regards

Jeff Zhang

[Jeff Zhang]

I know some users do the same thing in spark. Usually the service run spark driver side. But flink is different from spark. Spark driver is equal to flink client + flink job manager. I don't think currently we allow to run any user code in job manager. So allow running user defined service in job manager might a big change for flink.

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午11:06写道：

In my mind the user API could run everywhere but the simplest thing is to make them available in the Job Manager (where the other REST API lives).

They could become a very simple but powerful way to add valuable services to Flink without adding useless complexity to the overall architecture for just a few methods.

I don't know whether Spark or Beam allow you to do something like that but IMHO it's super useful (especially from a maintenance point of view wrt the overall architecture complexity).

@Oytun indeed we'd like to avoid recompiling everything when a single user class (i.e. not related to Flink classes) is modified or added. Glad to see that there are other people having the same problem here

On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <[hidden email]> wrote:

Hi Flavio,

I am curious know where service run, Do you create this service in UDF and run it  in TM ?

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午8:30写道：

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

--

Best Regards

Jeff Zhang

--

[Oytun Tez]

I had some back and forth last year about this, I'll forward the discussion email chain to you privately (it was in this mailing list).

Basically, the idea was to make DispatcherRestEndpoint and/or WebMonitorExtension more accessible so we can extend them. It didn't look too much work on Flink's side, but of course I can't tell confidently.

On Tue, Apr 21, 2020 at 11:12 AM Jeff Zhang <[hidden email]> wrote:

I know some users do the same thing in spark. Usually the service run spark driver side. But flink is different from spark. Spark driver is equal to flink client + flink job manager. I don't think currently we allow to run any user code in job manager. So allow running user defined service in job manager might a big change for flink.

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午11:06写道：

In my mind the user API could run everywhere but the simplest thing is to make them available in the Job Manager (where the other REST API lives).

They could become a very simple but powerful way to add valuable services to Flink without adding useless complexity to the overall architecture for just a few methods.

I don't know whether Spark or Beam allow you to do something like that but IMHO it's super useful (especially from a maintenance point of view wrt the overall architecture complexity).

@Oytun indeed we'd like to avoid recompiling everything when a single user class (i.e. not related to Flink classes) is modified or added. Glad to see that there are other people having the same problem here

On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <[hidden email]> wrote:

Hi Flavio,

I am curious know where service run, Do you create this service in UDF and run it  in TM ?

Flavio Pompermaier <[hidden email]> 于2020年4月21日周二 下午8:30写道：

Hi to all,

many times it happens that we use Flink as a broker towards the data layer but we need to be able to get some specific info from the data sources we use (i.e. get triggers and relationships from jdbc).

The quick and dirty way of achieving this is to run a Flink job that calls another service to store the required info. Another solution could be to add a custom REST service that contains a lot of dependencies already provided by Flink, with the risk of having misaligned versions between the 2..

It would be much simpler to enable users to add custom REST services to Flink in a configurable file. something like:

/myservice1/* -> com.example.MyRestService1

/myservice2/* -> com.example.MyRestService2

The listed classes should be contained in a jar within the Flink lib dir and should implement a common interface.

In order to avoid path collisions with already existing FLINK services, the configured path can be further prefixed with some other token (e.g. /userapi/*).

What do you think about this? Does it sound reasonable to you?

Am I the only one that thinks this could be useful for many use cases?

Best,

Flavio

--

Best Regards

Jeff Zhang

--

Best Regards

Jeff Zhang