Running Flink job as a rest

[dhurandar S]

Can Flink job be running as Rest Server, Where Apache Flink job is listening on a port (443). When a user calls this URL with payload, data directly goes to the Apache Flink windowing function.

Right now Flink can ingest data from Kafka or Kinesis, but we have a use case where we would like to push data to Flink, where Flink is listening on a port 

--

[Yun Tang]

Hi Dhurandar,

I'm afraid that Flink's rest API cannot satisfy your request as it would not act as any source. One possible example could be SocketWindowWordCount [1] which listens data on a port from all taskmanagers with sources.

[1] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java

Best

Yun Tang

---

From: dhurandar S <[hidden email]>
Sent: Thursday, December 3, 2020 5:31
To: Flink Dev <[hidden email]>; user <[hidden email]>
Subject: Running Flink job as a rest

 

Can Flink job be running as Rest Server, Where Apache Flink job is listening on a port (443). When a user calls this URL with payload, data directly goes to the Apache Flink windowing function.

Right now Flink can ingest data from Kafka or Kinesis, but we have a use case where we would like to push data to Flink, where Flink is listening on a port 

--

[Chesnay Schepler]

What you are asking for is an HTTP(s) source. This currently does not
exist for Flink, so you would have to implement it yourself.
Additionally you would have to figure out the host on which the source
runs on yourself.

It may be easier to setup a separate HTTP(s) server that accepts data,
which you then query from the source.

On 12/2/2020 10:31 PM, dhurandar S wrote:
> Can Flink job be running as Rest Server, Where Apache Flink job is
> listening on a port (443). When a user calls this URL with payload,
> data directly goes to the Apache Flink windowing function.
>
> Right now Flink can ingest data from Kafka or Kinesis, but we have a use
> case where we would like to push data to Flink, where Flink is listening on
> a port
>

[Jaffe, Julian]

I can't vouch for it personally, but perhaps the Apache Bahir Netty Source for Flink could help you? It sounds like you want to use HTTPS, which this doesn't support directly, but the source might be a helpful starting point to adding the functionality you need.

On 12/3/20, 1:33 AM, "Chesnay Schepler" <[hidden email]> wrote:

    What you are asking for is an HTTP(s) source. This currently does not
    exist for Flink, so you would have to implement it yourself.
    Additionally you would have to figure out the host on which the source
    runs on yourself.

    It may be easier to setup a separate HTTP(s) server that accepts data,
    which you then query from the source.

    On 12/2/2020 10:31 PM, dhurandar S wrote:
    > Can Flink job be running as Rest Server, Where Apache Flink job is
    > listening on a port (443). When a user calls this URL with payload,
    > data directly goes to the Apache Flink windowing function.
    >
    > Right now Flink can ingest data from Kafka or Kinesis, but we have a use
    > case where we would like to push data to Flink, where Flink is listening on
    > a port
    >