Extending REST API with new endpoints

[Oytun Tez]

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

[Oytun Tez]

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

[Biao Liu]

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

[Oytun Tez]

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

[Oytun Tez]

I did take a look at it, but things got out of hand very quickly from there on :D 

I see that WebSubmissionExtension implements WebMonitorExtension, but then WebSubmissionExtension was used in DispatcherRestEndpoint, which I couldn't know how to manipulate/extend...

How can I plug my Extension into the dispatcher?

On Mon, Jul 22, 2019 at 9:37 AM Seth Wiesman <[hidden email]> wrote:

Would the `WebMonitorExtension` work?

[1] https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorExtension.java

On Mon, Jul 22, 2019 at 8:35 AM Oytun Tez <[hidden email]> wrote:

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

--

[Oytun Tez]

Ping, any ideas?

On Mon, Jul 22, 2019 at 9:39 AM Oytun Tez <[hidden email]> wrote:

I did take a look at it, but things got out of hand very quickly from there on :D 

I see that WebSubmissionExtension implements WebMonitorExtension, but then WebSubmissionExtension was used in DispatcherRestEndpoint, which I couldn't know how to manipulate/extend...

How can I plug my Extension into the dispatcher?

On Mon, Jul 22, 2019 at 9:37 AM Seth Wiesman <[hidden email]> wrote:

Would the `WebMonitorExtension` work?

[1] https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorExtension.java

On Mon, Jul 22, 2019 at 8:35 AM Oytun Tez <[hidden email]> wrote:

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

--

[Chesnay Schepler]

There's no built-in way to extend the REST API. You will have to create a fork and either extend the DIspatcherRestEndpoint (or parent classes), or implement another WebMonitorExtension and modify the DispatcherRestEndpoint to load that one as well.

On 23/07/2019 15:51, Oytun Tez wrote:

Ping, any ideas?

On Mon, Jul 22, 2019 at 9:39 AM Oytun Tez <[hidden email]> wrote:

I did take a look at it, but things got out of hand very quickly from there on :D 

I see that WebSubmissionExtension implements WebMonitorExtension, but then WebSubmissionExtension was used in DispatcherRestEndpoint, which I couldn't know how to manipulate/extend...

How can I plug my Extension into the dispatcher?

On Mon, Jul 22, 2019 at 9:37 AM Seth Wiesman <[hidden email]> wrote:

Would the `WebMonitorExtension` work?

[1] https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorExtension.java

On Mon, Jul 22, 2019 at 8:35 AM Oytun Tez <[hidden email]> wrote:

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

--

[Oytun Tez]

Scary! :) I would heartily hate to maintain our own fork.

Should I make a feature request to discuss further and then send a PR for this? Is this the normal way to push for a feature?

On Fri, Jul 26, 2019 at 8:01 AM Chesnay Schepler <[hidden email]> wrote:

There's no built-in way to extend the REST API. You will have to create a fork and either extend the DIspatcherRestEndpoint (or parent classes), or implement another WebMonitorExtension and modify the DispatcherRestEndpoint to load that one as well.

On 23/07/2019 15:51, Oytun Tez wrote:

Ping, any ideas?

On Mon, Jul 22, 2019 at 9:39 AM Oytun Tez <[hidden email]> wrote:

I did take a look at it, but things got out of hand very quickly from there on :D 

I see that WebSubmissionExtension implements WebMonitorExtension, but then WebSubmissionExtension was used in DispatcherRestEndpoint, which I couldn't know how to manipulate/extend...

How can I plug my Extension into the dispatcher?

On Mon, Jul 22, 2019 at 9:37 AM Seth Wiesman <[hidden email]> wrote:

Would the `WebMonitorExtension` work?

[1] https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorExtension.java

On Mon, Jul 22, 2019 at 8:35 AM Oytun Tez <[hidden email]> wrote:

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

--

[Fabian Hueske-2]

Hi Oytun,

Thanks for your input and feature request!

The right way to propose a feature and contribute it is described here [1].

Basically, you should open a Jira issue and start a discussion about the feature there.

If it is a bigger features, you should also bring it to the [hidden email] mailing list.

Best,

Fabian

[1] https://flink.apache.org/contributing/contribute-code.html

Am Fr., 26. Juli 2019 um 16:03 Uhr schrieb Oytun Tez <[hidden email]>:

Scary! :) I would heartily hate to maintain our own fork.

Should I make a feature request to discuss further and then send a PR for this? Is this the normal way to push for a feature?

On Fri, Jul 26, 2019 at 8:01 AM Chesnay Schepler <[hidden email]> wrote:

There's no built-in way to extend the REST API. You will have to create a fork and either extend the DIspatcherRestEndpoint (or parent classes), or implement another WebMonitorExtension and modify the DispatcherRestEndpoint to load that one as well.

On 23/07/2019 15:51, Oytun Tez wrote:

Ping, any ideas?

On Mon, Jul 22, 2019 at 9:39 AM Oytun Tez <[hidden email]> wrote:

I did take a look at it, but things got out of hand very quickly from there on :D 

I see that WebSubmissionExtension implements WebMonitorExtension, but then WebSubmissionExtension was used in DispatcherRestEndpoint, which I couldn't know how to manipulate/extend...

How can I plug my Extension into the dispatcher?

On Mon, Jul 22, 2019 at 9:37 AM Seth Wiesman <[hidden email]> wrote:

Would the `WebMonitorExtension` work?

[1] https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorExtension.java

On Mon, Jul 22, 2019 at 8:35 AM Oytun Tez <[hidden email]> wrote:

I simply want to open up endpoints to query QueryableStates. What I had in mind was to give operators an interface to implement their own QueryableState controllers, e.g. serializers etc.

We are trying to use Flink in more of an "application framework" fashion, so extensibility helps a lot. As there already is a http server in this codebase, we'd like to attach to that instead. Especially queryable state is tightly coupled with Flink code, so it doesn't make much sense to host another http application to bridge into Flink.

On Mon, Jul 22, 2019 at 4:38 AM Biao Liu <[hidden email]> wrote:

Hi,

As far as I know, the RESTful handler is not pluggable. And I don't see a strong reason from your description to do so.

Could you explain more about your requirement? 

Oytun Tez <[hidden email]> 于2019年7月20日周六 上午4:36写道：

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in.

I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster.

Any pointers?

On Fri, Jul 19, 2019 at 3:53 PM Oytun Tez <[hidden email]> wrote:

Hi there,

I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside.

Can I do this with 1.8? Any other way or plans to make this possible?

--