Flink's status for Unbounded source goes to Finish after upgrade to 1.10

[Akshay Iyangar]

Hi

 

We recently decided to upgrade our Flink cluster from 1.8 to 1.10.

 

Post the upgrade we see issues where our pipelines which read data from unbounded sources (kinesis) go to FINISH state. I’m a little confused to see why that would be happening especially if it is an unbounded source?

 

Also to give some context mostly all the pipelines are written in Beam and run on Flink.

 

 

SNAPS

FLINK 1.8

 

 

Flink 1.10

 

 

Any idea why this might be happening?

 

 

[Till Rohrmann]

Hi Akshay,

could you check the Flink logs and in particular the TaskManager logs where the source operators are running. They might contain some clue why the sources believe that they can stop. If the logs are not conclusive, then you could also enable the DEBUG log level for more information. You can also share the logs with us for further debugging.

Cheers,

Till

On Fri, Jun 12, 2020 at 6:27 AM Akshay Iyangar <[hidden email]> wrote:

Hi

 

We recently decided to upgrade our Flink cluster from 1.8 to 1.10.

 

Post the upgrade we see issues where our pipelines which read data from unbounded sources (kinesis) go to FINISH state. I’m a little confused to see why that would be happening especially if it is an unbounded source?

 

Also to give some context mostly all the pipelines are written in Beam and run on Flink.

 

 

SNAPS

FLINK 1.8

 

 

Flink 1.10

 

 

Any idea why this might be happening?