Is a flink-1.11.2 compiled job run on flink-1.11.0 cluster?

[Marco Villalobos-2]

SITUATION

Amazon EMR 6.1 supports Flink 1.11.0 and JDK 11.

Docker Flink images support for Flink 1.11.0 only supports JDK 8.

Much of the testing that I perform uses Docker and Docker compose.

It's much easier for me to develop with Docker with Flink 1.11.2 and JDK 11.  

QUESTION

Would a flink-1.11.2 compiled job run on flink-1.11.0 cluster?

I could not find any notes on backwards compatibility of minor versions.

[rmetzger0]

Hi,

I agree that our docs are not mentioning this anywhere. I would have expected it on this page: https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html. I filed a ticket to address this: https://issues.apache.org/jira/browse/FLINK-19955

The only thing Flink officially guarantees is that a job compiled against 1.11.0 also compiles against 1.11.2. There could be protocol or format changes that could break if you are using a Flink job compiled against an older version.

However, chances are high that the job will be effectively compatible with the 1.11.0 cluster, because we are not changing that much between minor releases.

On Tue, Nov 3, 2020 at 2:48 PM Marco Villalobos <[hidden email]> wrote:

SITUATION

Amazon EMR 6.1 supports Flink 1.11.0 and JDK 11.

Docker Flink images support for Flink 1.11.0 only supports JDK 8.

Much of the testing that I perform uses Docker and Docker compose.

It's much easier for me to develop with Docker with Flink 1.11.2 and JDK 11. 

QUESTION

Would a flink-1.11.2 compiled job run on flink-1.11.0 cluster?

I could not find any notes on backwards compatibility of minor versions.

[Arvid Heise-3]

Hi Marco,

to add to Robert's answer. Minor releases are mostly and foremost used as bugfixes (actually from semantic versioning guideline, the last digit indicates a bugfix release). So there are no API changes between 1.11.0 and 1.11.2. However, sometimes bugfixes require some changes of underlying code. So there is still a bit of a risk. But it's definitely intended that your use case works.

Please note that by nature of bugfix releases, some things that work in 1.11.2 do not work in 1.11.0 (out of the box). So while a given job in 1.11.2 works for you locally, it could fail on EMR because of some known bug. At this point, there is not much you can do if you insist on running on EMR except waiting for EMR to catch up.

For new deployments, I always recommend to use a K8s based solution instead of any cloud vendor specific stuff, which is especially viable for you as you already use Docker.

On Tue, Nov 3, 2020 at 5:10 PM Robert Metzger <[hidden email]> wrote:

Hi,

I agree that our docs are not mentioning this anywhere. I would have expected it on this page: https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html. I filed a ticket to address this: https://issues.apache.org/jira/browse/FLINK-19955

The only thing Flink officially guarantees is that a job compiled against 1.11.0 also compiles against 1.11.2. There could be protocol or format changes that could break if you are using a Flink job compiled against an older version.

However, chances are high that the job will be effectively compatible with the 1.11.0 cluster, because we are not changing that much between minor releases.

On Tue, Nov 3, 2020 at 2:48 PM Marco Villalobos <[hidden email]> wrote:

SITUATION

Amazon EMR 6.1 supports Flink 1.11.0 and JDK 11.

Docker Flink images support for Flink 1.11.0 only supports JDK 8.

Much of the testing that I perform uses Docker and Docker compose.

It's much easier for me to develop with Docker with Flink 1.11.2 and JDK 11. 

QUESTION

Would a flink-1.11.2 compiled job run on flink-1.11.0 cluster?

I could not find any notes on backwards compatibility of minor versions.

--