TCP streams to multiple clients

[Simec, Nick]

I’d like to put flink on a proxy server to read in a stream from an external source and then distribute that stream to multiple servers. Is that possible with Flink? Would I have to replicate the data or anything?

 

 

I want source -> proxy-server -> (FLINK) -> -> -> -> -> 5 servers

 

Flink reads the stream coming in to the proxy-server and sends it to 5 other servers.

[rmetzger0]

Hey Nick,

Yes, Flink is able to send data to many destinations (we call them sinks). You don't need to manually replicate the data.

In pseudocode, it would look like this:

DataStream<Data> stream = env.source(ProxyReader())

// potentially do data processing such as filtering, cleaning, windowing, .. on "stream"

stream.addSink(TCPStreamer("dest1"))

stream.addSink(TCPStreamer("dest2"))

stream.addSink(TCPStreamer("dest3"))

stream.addSink(KafkaProducer)

Each sink would write the data from the stream to its destination.

Flink comes with a number of predefined sources and sinks for commonly used systems, but it is also fairly straightforward to build your own (if you want to make them fault tolerant, it'll be trickier)

Have you considered using Apache Kafka (or Amazon Kinesis, Google Pubsub or any message queue) for sending data between these systems?

Usually, using something like Kafka gives you a lot of benefits for fault tolerance and buffering data between services.

Best,

Robert

On Tue, Apr 7, 2020 at 5:46 PM Simec, Nick <[hidden email]> wrote:

I’d like to put flink on a proxy server to read in a stream from an external source and then distribute that stream to multiple servers. Is that possible with Flink? Would I have to replicate the data or anything?

 

 

I want source -> proxy-server -> (FLINK) -> -> -> -> -> 5 servers

 

Flink reads the stream coming in to the proxy-server and sends it to 5 other servers.