Multiple restarts of Local Cluster

[Sachin Goel]

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

[Stephan Ewen]

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Sachin Goel]

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Till Rohrmann]

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Sachin Goel]

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Till Rohrmann]

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Till Rohrmann]

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Sachin Goel]

Okay. No problem.

Any suggestions for the correct context though? :')

I don't think something like a {{FlinkProgram}} class is a good idea [User would need to override a {{program}} method and we will make sure the cluster is setup only once and torn down properly only after the user code finishes completely].

However, if this isn't so, the shutting down of cluster becomes impossible.

Can I assume however, that the actor system will be shut down automatically when the main method exits? After all the JVM will terminate. If so, I can make some changes in LocalExecutor to start up the cluster only once.

Regards

Sachin

On Wed, Sep 2, 2015 at 9:33 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Sachin Goel]

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Till Rohrmann-2]

If I'm not mistaken, then the cluster should be properly terminated when it gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <[hidden email]> wrote:

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Stephan Ewen]

You can always shut down a cluster manually (via shutdown()) and if the JVM simply exists, all is well as well. Crucial cleanup is in shutdown hooks.

On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann <[hidden email]> wrote:

If I'm not mistaken, then the cluster should be properly terminated when it gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <[hidden email]> wrote:

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Sachin Goel]

I'm not sure what you mean by "Crucial cleanup is in shutdown hooks". Could you elaborate?

On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen <[hidden email]> wrote:

You can always shut down a cluster manually (via shutdown()) and if the JVM simply exists, all is well as well. Crucial cleanup is in shutdown hooks.

On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann <[hidden email]> wrote:

If I'm not mistaken, then the cluster should be properly terminated when it gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <[hidden email]> wrote:

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Stephan Ewen]

Stopping the JVM process clean up all resources, except temp files.

Everything that creates temp files uses a shutdown hook to remove these: IOManager, BlobManager, LibraryCache, ...

On Wed, Sep 2, 2015 at 7:40 PM, Sachin Goel <[hidden email]> wrote:

I'm not sure what you mean by "Crucial cleanup is in shutdown hooks". Could you elaborate?

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen <[hidden email]> wrote:

You can always shut down a cluster manually (via shutdown()) and if the JVM simply exists, all is well as well. Crucial cleanup is in shutdown hooks.

On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann <[hidden email]> wrote:

If I'm not mistaken, then the cluster should be properly terminated when it gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <[hidden email]> wrote:

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

[Stephan Ewen]

Have a look at the class IOManager and IOManagerAsync, it is a good example of how we use these hooks for cleanup.

The constructor usually installs them, and the shutdown logic removes them.

On Thu, Sep 3, 2015 at 9:19 PM, Stephan Ewen <[hidden email]> wrote:

Stopping the JVM process clean up all resources, except temp files.

Everything that creates temp files uses a shutdown hook to remove these: IOManager, BlobManager, LibraryCache, ...

On Wed, Sep 2, 2015 at 7:40 PM, Sachin Goel <[hidden email]> wrote:

I'm not sure what you mean by "Crucial cleanup is in shutdown hooks". Could you elaborate?

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen <[hidden email]> wrote:

You can always shut down a cluster manually (via shutdown()) and if the JVM simply exists, all is well as well. Crucial cleanup is in shutdown hooks.

On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann <[hidden email]> wrote:

If I'm not mistaken, then the cluster should be properly terminated when it gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <[hidden email]> wrote:

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway.

I will make sure if this is the case.

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <[hidden email]> wrote:

Maybe we can create a single PlanExecutor for the LocalEnvironment which is used when calling execute. This of course entails that we don’t call stop on the LocalCluster. For cases where the program exits after calling execute, this should be fine because all resources will then be released anyway. It might matter for the test execution where maven reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected right away. You could try it out and see what happens.

Cheers,
Till

​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <[hidden email]> wrote:

Oh sorry, then I got the wrong context. I somehow thought it was about test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <[hidden email]> wrote:

I was under the impression that the @AfterClass annotation can only be used in test classes.

Even so, the idea is that a user program running in the IDE should not be starting up the cluster several times [my primary concern is the addition of the persist operator], and we certainly cannot ask the user to terminate the cluster after execution, while in local mode. 

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <[hidden email]> wrote:

Why is it not possible to shut down the local cluster? Can’t you shut it down in the @AfterClass method?

​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <[hidden email]> wrote:

Yes. That will work too. However, then it isn't possible to shut down the local cluster. [Is it necessary to do so or does it shut down automatically when the program exists? I'm not entirely sure.]

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <[hidden email]> wrote:

Have a look at some other tests, like the checkpointing tests. They start one cluster manually and keep it running. They connect against it using the remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <[hidden email]> wrote:

Hi all

While using LocalEnvironment, in case the program triggers execution several times, the {{LocalFlinkMiniCluster}} is started as many times. This can consume a lot of time in setting up and tearing down the cluster. Further, this hinders with a new functionality I'm working on based on persisted results.

One potential solution could be to follow the methodology in `MultipleProgramsTestBase`. The user code then would have to reside in a fixed function name, instead of the main method. Or is that too cumbersome?

Regards

Sachin

-- Sachin Goel

Computer Science, IIT Delhi

m. <a href="tel:%2B91-9871457685" value="[phone]" target="_blank">[phone]