Application Mode support on VVP v2.3

[narasimha]

Hi,

Using ververica platform to deploy flink jobs, found that it is not supporting application deployment mode. 

Just want to check if it is expected. 

Below is a brief  of how the main method has been composed. 

class Job1 {

    public void execute(){

        StreamExecutingEnvironemnt env = ...

        env.execute("Job 1");

    }

}

class Job2 {

    public void execute(){

        StreamExecutingEnvironemnt env = ...

        env.execute("Job 2");

    }

}

class Main {

    public static void main(string[] args) {

    new Job1().execute();

    new Job2().execute();

    }

}

Does this make any difference?

TIA

--

[Till Rohrmann]

Hi Narasimha,

thanks for reaching out to the community. I am not entirely sure whether VVP 2.3 supports the application mode. Since this is a rather new feature, it could be that it has not been integrated yet. I am pulling in Ufuk and Fabian who should be able to definitely answer your question. Maybe they can also tell whether this is planned to be changed in the foreseeable future.

Cheers,

Till

On Wed, Dec 2, 2020 at 4:33 PM narasimha <[hidden email]> wrote:

Hi,

Using ververica platform to deploy flink jobs, found that it is not supporting application deployment mode. 

Just want to check if it is expected. 

Below is a brief  of how the main method has been composed. 

class Job1 {

    public void execute(){

        StreamExecutingEnvironemnt env = ...

        env.execute("Job 1");

    }

}

class Job2 {

    public void execute(){

        StreamExecutingEnvironemnt env = ...

        env.execute("Job 2");

    }

}

class Main {

    public static void main(string[] args) {

    new Job1().execute();

    new Job2().execute();

    }

}

Does this make any difference?

TIA

--

A.Narasimha Swamy

[Fabian Paul]

Hi Narasimha,

Nothing comes to my mind immediately why it should not work. We are using the StandaloneApplicationClusterEntryPoint to start the cluster. Can you provide some more information about which Flink image on vvp are you trying to use and maybe show the error message?

Best,
Fabian

[narasimha]

thanks Fabian for responding.

flink image : registry.ververica.com/v2.2/flink:1.11.1-stream1-scala_2.12

There are no errors as such. But it is just considering the first job.

On Thu, Dec 3, 2020 at 5:34 PM Fabian Paul <[hidden email]> wrote:

Hi Narasimha,

Nothing comes to my mind immediately why it should not work. We are using the StandaloneApplicationClusterEntryPoint to start the cluster. Can you provide some more information about which Flink image on vvp are you trying to use and maybe show the error message?

Best,
Fabian

--

[Fabian Paul]

Hi Narasimha,

I investigated your problem and it is caused by multiple issues. First vvp in

general cannot really handle multi job submissions per jar because the complete

deployment lifecycle in vvp is scoped around a single Flink job id.

Therefore vvp sets a generated Flink job id during submission which leads to

that only a single job is executed [1].

Secondly in your example you are using `execute()` on the StreamEnvironment

which is a blocking operation. You need to use `executeAsync()` to execute the

jobs simultanously.

Best,

Fabian

[1] https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L178

On 6. Dec 2020, at 15:10, narasimha <[hidden email]> wrote:

thanks Fabian for responding.

flink image : registry.ververica.com/v2.2/flink:1.11.1-stream1-scala_2.12

There are no errors as such. But it is just considering the first job.

On Thu, Dec 3, 2020 at 5:34 PM Fabian Paul <[hidden email]> wrote:

Hi Narasimha,

Nothing comes to my mind immediately why it should not work. We are using the StandaloneApplicationClusterEntryPoint to start the cluster. Can you provide some more information about which Flink image on vvp are you trying to use and maybe show the error message?

Best,
Fabian

--

[narasimha]

Thanks for the information.

Are there any plans to implement this? It is supported on other docker images...

On Tue, 8 Dec 2020 at 9:36 PM, Fabian Paul <[hidden email]> wrote:

Hi Narasimha,

I investigated your problem and it is caused by multiple issues. First vvp in

general cannot really handle multi job submissions per jar because the complete

deployment lifecycle in vvp is scoped around a single Flink job id.

Therefore vvp sets a generated Flink job id during submission which leads to

that only a single job is executed [1].

Secondly in your example you are using `execute()` on the StreamEnvironment

which is a blocking operation. You need to use `executeAsync()` to execute the

jobs simultanously.

Best,

Fabian

[1] https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L178

On 6. Dec 2020, at 15:10, narasimha <[hidden email]> wrote:

thanks Fabian for responding.

flink image : registry.ververica.com/v2.2/flink:1.11.1-stream1-scala_2.12

There are no errors as such. But it is just considering the first job.

On Thu, Dec 3, 2020 at 5:34 PM Fabian Paul <[hidden email]> wrote:

Hi Narasimha,

Nothing comes to my mind immediately why it should not work. We are using the StandaloneApplicationClusterEntryPoint to start the cluster. Can you provide some more information about which Flink image on vvp are you trying to use and maybe show the error message?

Best,
Fabian

--

A.Narasimha Swamy

--

[Fabian Paul]

Hi Narasimha,

I think without a major change in vvp it will not be possible to submit
multiple jobs within one jar in the foreseeable future. Maybe you can explain
more about your use case why it is inconvenient for you to put the jobs in
multiple jars?
With ververica platform 2.4 we will add support for deployments
on session clusters, do you think it would help in your case?

Best,
Fabian