Apache Flink - Multiple Kinesis stream consumers

[M Singh]

Hi:

I am trying to understand how does Flink coordinate multiple kinesis consumers and am trying to find out:

1. Is it possible to read same kinesis stream independently multiple times within a single application ? How does Flink coordinate consuming same kinesis multiple times in a single application ? Are there any issues that can arise from this pattern ?

2. How does Flink coordinate consumers of same kinesis stream across multiple applications ? 

Thanks

Mans

[Konstantin Knauf-2]

Hi Mans,

Flink does not coordinate the consumption at all (neither within the same nor across applications). Each FlinkKinesisConsumer will keep track of its consumption (position in each shared) in order to provide exactly-once guarantees.

Cheers,

Konstantin

On Mon, Jul 1, 2019 at 12:30 PM M Singh <[hidden email]> wrote:

Hi:

I am trying to understand how does Flink coordinate multiple kinesis consumers and am trying to find out:

1. Is it possible to read same kinesis stream independently multiple times within a single application ? How does Flink coordinate consuming same kinesis multiple times in a single application ? Are there any issues that can arise from this pattern ?

2. How does Flink coordinate consumers of same kinesis stream across multiple applications ? 

Thanks

Mans

--