run flink on edge vs hub

[Eleanore Jin]

Hi Community, 

Currently we are running flink in 'hub' data centers where data is ingested into the platform via kafka, and flink job will read from kafka, do the transformations, and publish to another kafka topic. 

I would also like to see if the same logic (read input message -> do transformation -> return output message) can be applied on 'edge' data centers. 

The requirement for run on 'edge' is to return the response synchronously. Like the synchronous http based request/response. 

Can you please provide some guidance/thoughts on this? 

Thanks a lot!

Eleanore

[Arvid Heise-3]

Hi Eleanore,

The question in general is what you understand under edge data centers as the term is pretty fuzzy. Since Flink is running on Java, it's not suitable for embedded clusters as of now. There is plenty of work done already to tests that Flink runs on ARM clusters [1].

If you just mean in general moving away from a monolithic hub cluster to smaller clusters, then this is easily done with Flink on the compute side. The question is rather how data storage should look in such an edge setting and how the interfaces look.

From your example, it seems as if you want to use Flink as a reactive server, possibly easily scalable. If so, then yes it is possible with Flink, even though I'd say it's not the primary use case for Flink. In any case, synchronous requests will be a bit difficult/unnatural. I'd probably go for an async job pattern. So Flink listens to some port for requests (socketTextStream [2]) with a job id, processes data and keeps the data in state keyed by job id. The client then uses the job id to fetch the job state through queryable state [2]. The responses eventually time out through TTL [4].

Of course, you'd put a small proxy in front of that composited job (separate input/query port) that translates the queries from the client to the Flink job. The proxy would most likely also generate the job id and return it to the client. Ultimately, that proxy could offer a synchronous interface and pull for the result itself, but that makes the proxy suddenly quite heavy.

The proxy setup can be reused for different edge clusters making it a one time investment. Note that there are other software stacks for reactive servers that offer the functionality out of the box.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-ARM-support-for-Flink-td30298.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/datastream_api.html#data-sources

[3] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html

[4] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl

On Mon, May 18, 2020 at 4:39 AM Eleanore Jin <[hidden email]> wrote:

Hi Community, 

Currently we are running flink in 'hub' data centers where data is ingested into the platform via kafka, and flink job will read from kafka, do the transformations, and publish to another kafka topic. 

I would also like to see if the same logic (read input message -> do transformation -> return output message) can be applied on 'edge' data centers. 

The requirement for run on 'edge' is to return the response synchronously. Like the synchronous http based request/response. 

Can you please provide some guidance/thoughts on this? 

Thanks a lot!

Eleanore

--

[Eleanore Jin]

Hi Arvid, 

Thanks for the suggestion! I will tryout to see how it works. 

Best,

Eleanore

On Mon, May 18, 2020 at 8:04 AM Arvid Heise <[hidden email]> wrote:

Hi Eleanore,

The question in general is what you understand under edge data centers as the term is pretty fuzzy. Since Flink is running on Java, it's not suitable for embedded clusters as of now. There is plenty of work done already to tests that Flink runs on ARM clusters [1].

If you just mean in general moving away from a monolithic hub cluster to smaller clusters, then this is easily done with Flink on the compute side. The question is rather how data storage should look in such an edge setting and how the interfaces look.

From your example, it seems as if you want to use Flink as a reactive server, possibly easily scalable. If so, then yes it is possible with Flink, even though I'd say it's not the primary use case for Flink. In any case, synchronous requests will be a bit difficult/unnatural. I'd probably go for an async job pattern. So Flink listens to some port for requests (socketTextStream [2]) with a job id, processes data and keeps the data in state keyed by job id. The client then uses the job id to fetch the job state through queryable state [2]. The responses eventually time out through TTL [4].

Of course, you'd put a small proxy in front of that composited job (separate input/query port) that translates the queries from the client to the Flink job. The proxy would most likely also generate the job id and return it to the client. Ultimately, that proxy could offer a synchronous interface and pull for the result itself, but that makes the proxy suddenly quite heavy.

The proxy setup can be reused for different edge clusters making it a one time investment. Note that there are other software stacks for reactive servers that offer the functionality out of the box.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-ARM-support-for-Flink-td30298.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/datastream_api.html#data-sources

[3] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html

[4] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl

On Mon, May 18, 2020 at 4:39 AM Eleanore Jin <[hidden email]> wrote:

Hi Community, 

Currently we are running flink in 'hub' data centers where data is ingested into the platform via kafka, and flink job will read from kafka, do the transformations, and publish to another kafka topic. 

I would also like to see if the same logic (read input message -> do transformation -> return output message) can be applied on 'edge' data centers. 

The requirement for run on 'edge' is to return the response synchronously. Like the synchronous http based request/response. 

Can you please provide some guidance/thoughts on this? 

Thanks a lot!

Eleanore

--

Arvid Heise | Senior Java Developer

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng