Correct way to e2e test a Flink application?

[Laurent Exsteens]

Hello,

I would like to test a Flink application, including any problem that would happen when deployed on a distributed cluster.

The way we do this currently is to launch a Flink cluster in Docker and run the job on it. This setup seems heavy and might not be necessary.

Is there a way to simulate a Flink cluster as if it would be distributed (including serialisation and static classes behavior on such a cluster) using, for example, a Flink Mini cluster? If yes, is there any resources available on how to setup a correct test environment?

Thanks in advance for your help.

Regards,

Laurent.

♻ Be green, keep it on the screen

[Tzu-Li (Gordon) Tai]

Hi Laurent,

You can take a look at Flink's MiniClusterResource JUnit test rule, and its
usages in the codebase for that.
The rule launches a Flink MiniCluster within the same JVM, and submission to
the mini cluster resembles how it would be submitting to an actual Flink
cluster, so you would already be able to catch problems such as operator
serialization errors.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Laurent Exsteens]

Hi Tzu-Li,

thanks a lot for your answer. I will try this!

However, I was looking for something that does fully simulate a Flink cluster, including job-manager to task manager serialization issues and full isolation of the task managers (I guess in the MiniClusterResource, we are still on the same JVM, so static classes will still be available to all operators, which is different than on a real cluster).

Cheers,

Laurent.

On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Laurent,

You can take a look at Flink's MiniClusterResource JUnit test rule, and its
usages in the codebase for that.
The rule launches a Flink MiniCluster within the same JVM, and submission to
the mini cluster resembles how it would be submitting to an actual Flink
cluster, so you would already be able to catch problems such as operator
serialization errors.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--

Laurent Exsteens

Data Engineer

(M) [phone]

EURA NOVA

Rue Emile Francqui, 4

1435 Mont-Saint-Guibert

(T) <a href="tel:%2B32%2010%2075%2002%2000" value="[phone]" style="color:rgb(17,85,204)" target="_blank">[phone]

euranova.eu

research.euranova.eu

♻ Be green, keep it on the screen

[rmetzger0]

Hey Laurent,

Flink developed an internal framework for executing end to end tests from Java. Here's an example of such a test: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java#L98

This should download and set up a real, distributed (on your machine, but in different processes) Flink cluster.

Best,

Robert

On Tue, Mar 31, 2020 at 10:48 AM Laurent Exsteens <[hidden email]> wrote:

Hi Tzu-Li,

thanks a lot for your answer. I will try this!

However, I was looking for something that does fully simulate a Flink cluster, including job-manager to task manager serialization issues and full isolation of the task managers (I guess in the MiniClusterResource, we are still on the same JVM, so static classes will still be available to all operators, which is different than on a real cluster).

Cheers,

Laurent.

On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Laurent,

You can take a look at Flink's MiniClusterResource JUnit test rule, and its
usages in the codebase for that.
The rule launches a Flink MiniCluster within the same JVM, and submission to
the mini cluster resembles how it would be submitting to an actual Flink
cluster, so you would already be able to catch problems such as operator
serialization errors.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--

Laurent Exsteens

Data Engineer

(M) [phone]

EURA NOVA

Rue Emile Francqui, 4

1435 Mont-Saint-Guibert

(T) <a href="tel:%2B32%2010%2075%2002%2000" value="[phone]" style="color:rgb(17,85,204)" target="_blank">[phone]

euranova.eu

research.euranova.eu

♻ Be green, keep it on the screen

[Laurent Exsteens]

Hello Robert,

thanks a lot, I'll sure have a look at it!

Regards,

Laurent.

On Thu, 2 Apr 2020 at 10:54, Robert Metzger <[hidden email]> wrote:

Hey Laurent,

Flink developed an internal framework for executing end to end tests from Java. Here's an example of such a test: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java#L98

This should download and set up a real, distributed (on your machine, but in different processes) Flink cluster.

Best,

Robert

On Tue, Mar 31, 2020 at 10:48 AM Laurent Exsteens <[hidden email]> wrote:

Hi Tzu-Li,

thanks a lot for your answer. I will try this!

However, I was looking for something that does fully simulate a Flink cluster, including job-manager to task manager serialization issues and full isolation of the task managers (I guess in the MiniClusterResource, we are still on the same JVM, so static classes will still be available to all operators, which is different than on a real cluster).

Cheers,

Laurent.

On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Laurent,

You can take a look at Flink's MiniClusterResource JUnit test rule, and its
usages in the codebase for that.
The rule launches a Flink MiniCluster within the same JVM, and submission to
the mini cluster resembles how it would be submitting to an actual Flink
cluster, so you would already be able to catch problems such as operator
serialization errors.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--

Laurent Exsteens

Data Engineer

(M) [phone]

EURA NOVA

Rue Emile Francqui, 4

1435 Mont-Saint-Guibert

(T) <a href="tel:%2B32%2010%2075%2002%2000" value="[phone]" style="color:rgb(17,85,204)" target="_blank">[phone]

euranova.eu

research.euranova.eu

♻ Be green, keep it on the screen

--

♻ Be green, keep it on the screen