Re: Does anyone have an example of Bazel working with Flink?

Posted by Dan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Does-anyone-have-an-example-of-Bazel-working-with-Flink-tp35898p35972.html

Thanks for the replies!  I was able to use the provided answers to get a setup working (maybe not the most efficiently).  The main change I made was to switch to including the deploy jar in the image (rather than the default one).

I'm open to contributing to a "rules_flink" project.  I don't know enough yet to help design it.

On Sat, Jun 13, 2020 at 4:39 AM Till Rohrmann <[hidden email]> wrote:

Hi Dan,

if you want to run a Flink job without specifying the main class via `bin/flink run --class org.a.b.Foobar` then you have to add a MANIFEST.MF file to your jar under META-INF and this file needs to contain `Main-Class: org.a.b.Foobar`.

Cheers,

Till

On Fri, Jun 12, 2020 at 12:30 AM Austin Cawley-Edwards <[hidden email]> wrote:

Hey all,

Adding to Aaron's response, we use Bazel to build our Flink apps. We've open-sourced some of our setup here[1] though a bit outdated. There are definitely rough edges/ probably needs a good deal of work to fit other setups. We have written a wrapper around the `java_library` and `java_binary` and could do the same for `rules_scala`, though we just started using Bazel last November and have a lot to learn in terms of best practices there.

If you're interested in contributing to a `rules_flink` project, I would be as well!

Best,

Austin

[1]: https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020

On Thu, Jun 11, 2020 at 6:14 PM Aaron Levin <[hidden email]> wrote:

Hi Dan, 

We use Bazel to compile our Flink applications. We're using "rules_scala" (https://github.com/bazelbuild/rules_scala) to manage the dependencies and produce jars. We haven't had any issues. However, I have found that sometimes it's difficult to figure out exactly what Flink target or dependency my application needs.

Unfortunately I'm not sure what issue you're seeing here. I would guess either your flink application wasn't compiled into the jar you're executing. If you can paste the bazel target used to generate your jar and how you're launching the application, that will be helpful for diagnosis.

On Thu, Jun 11, 2020 at 5:21 PM Dan Hill <[hidden email]> wrote:

I took the Flink playground and I'm trying to swap out Maven for Bazel.  I got to the point where I'm hitting the following error.  I want to diff my code with an existing, working setup.

Thanks! - Dan

client_1                | org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.

client_1                | at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:596)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:190)

client_1                | at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:128)

client_1                | at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:862)

client_1                | at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:204)

client_1                | at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)

client_1                | at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)

client_1                | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)

client_1                | at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)