Re: Signal for End of Stream

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Signal-for-End-of-Stream-tp20002p20021.html

Hi,

Flink will automatically stop the execution of a DataStream program once all sources have finished to provide data, i.e., when all SourceFunction return from the run() method.

The DeserializationSchema.isEndOfStream() method can be used to tell a built-in SourceFunction such as a KafkaConsumer that it should leave the run() method.

If you implement your own SourceFunction you can leave run() after you ingested all data.

Note, that Flink won't wait for all processing time timers but will immediately shutdown the program after the last in-flight record was processed.
Event-time timers will be handled because each source emits a Long.MAX_VALUE watermark after it emitted its last record.

Best, Fabian

2018-05-07 17:18 GMT+02:00 Dhruv Kumar <[hidden email]>:

I notice that there is some DeserializationSchema in org.apache.flink.api.common.serialization which has a function isEndOfStream but I am not sure if I can use it in my use case. 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 7, 2018, at 06:18, Dhruv Kumar <[hidden email]> wrote:

Hi

Is there a way I can capture the end of stream signal for streams which are replayed from historical data? I need the end of stream signal to tell the Flink program to finish its execution.

Below is the use case in detail:

1. An independent log replayer program sends the records to a socket (identified by ip address and port).

2. Flink program reads the incoming records via socketTextStream from the above mentioned socket, applies a KeyBy operator on the incoming records and then does some processing, finally writing them to another socket.

How do I tell the Flink program to finish its execution? Is there any information which I can add to the records while they are sent from the replayer program and which can be parsed when the records arrive inside the Flink program?

Let me know if anything is not clear.

Thanks

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me