Does anyone have an example of Bazel working with Flink?

[Dan]

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[Aaron Levin]

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[austin.ce]

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[Till Rohrmann]

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[Dan]

Thanks for the replies!  I was able to use the provided answers to get a setup working (maybe not the most efficiently).  The main change I made was to switch to including the deploy jar in the image (rather than the default one).

I'm open to contributing to a "rules_flink" project.  I don't know enough yet to help design it.

On Sat, Jun 13, 2020 at 4:39 AM Till Rohrmann <[hidden email]> wrote:

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[austin.ce]

Great to hear Dan!

@Aaron - would you/ your team be interested in a `rules_flink` project? I'm still fairly new to Bazel and know enough to contribute, but could definitely use guidance on design as well.

Best,

Austin

On Mon, Jun 15, 2020 at 11:07 PM Dan Hill <[hidden email]> wrote:

Thanks for the replies!  I was able to use the provided answers to get a setup working (maybe not the most efficiently).  The main change I made was to switch to including the deploy jar in the image (rather than the default one).

I'm open to contributing to a "rules_flink" project.  I don't know enough yet to help design it.

On Sat, Jun 13, 2020 at 4:39 AM Till Rohrmann <[hidden email]> wrote:

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[Aaron Levin]

Hi Austin,

In our experience, `rules_scala` and `rules_java` are enough for us at this point. 

It's entirely possible I'm not thinking far enough into the future, though, so don't take our lack of investment as a sign it's not worth investing in :)

Best,

Aaron Levin

On Thu, Jun 18, 2020 at 10:27 AM Austin Cawley-Edwards <[hidden email]> wrote:

Great to hear Dan!

@Aaron - would you/ your team be interested in a `rules_flink` project? I'm still fairly new to Bazel and know enough to contribute, but could definitely use guidance on design as well.

Best,

Austin

On Mon, Jun 15, 2020 at 11:07 PM Dan Hill <[hidden email]> wrote:

Thanks for the replies!  I was able to use the provided answers to get a setup working (maybe not the most efficiently).  The main change I made was to switch to including the deploy jar in the image (rather than the default one).

I'm open to contributing to a "rules_flink" project.  I don't know enough yet to help design it.

On Sat, Jun 13, 2020 at 4:39 AM Till Rohrmann <[hidden email]> wrote:

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)

[austin.ce]

Ok, no worries Aaron, that's still good advice :)

One last question - are you using JAR-based or image-based deployments? The only real problem using Flink & Bazel and a JAR-based deployment from our experience is removing the Flink libs present in the deploy environment from the deploy jar, and still having them present when we want to do local debugging/ integration testing. Also possible that we're just not using Bazel entirely correctly!

Thank you!

Austin

On Thu, Jun 18, 2020 at 12:32 PM Aaron Levin <[hidden email]> wrote:

Hi Austin,

In our experience, `rules_scala` and `rules_java` are enough for us at this point. 

It's entirely possible I'm not thinking far enough into the future, though, so don't take our lack of investment as a sign it's not worth investing in :)

Best,

Aaron Levin

On Thu, Jun 18, 2020 at 10:27 AM Austin Cawley-Edwards <[hidden email]> wrote:

Great to hear Dan!

@Aaron - would you/ your team be interested in a `rules_flink` project? I'm still fairly new to Bazel and know enough to contribute, but could definitely use guidance on design as well.

Best,

Austin

On Mon, Jun 15, 2020 at 11:07 PM Dan Hill <[hidden email]> wrote:

Thanks for the replies!  I was able to use the provided answers to get a setup working (maybe not the most efficiently).  The main change I made was to switch to including the deploy jar in the image (rather than the default one).

I'm open to contributing to a "rules_flink" project.  I don't know enough yet to help design it.

On Sat, Jun 13, 2020 at 4:39 AM Till Rohrmann <[hidden email]> wrote:

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)