(DEPRECATED) Apache Flink User Mailing List archive.

Running StreamingJob.java in apache flink workflow

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
Abdullah bin Omar
Reply | Threaded
Open this post in threaded view
♦
♦
|

Running StreamingJob.java in apache flink workflow

Abdullah bin Omar
Hi,

I have just created a maven project in eclipse IDE by using a  sample code (named by StramingJob.java) that is under flink-quickstart-java (this file is one of the source files on flink). See the attached pic. 

According to quickstarts, I used the "mvn clean package" command and got a jar file [1] 

So, the next step is to run [2] in flink UI by using the command:
./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar

Question:
However, I have got some exceptions. This is the reason not to connect the jar file in the apache flink UI.

What is the solution in this case?



###########[1] Building jar from terminal ####

abdullah@Abdullahs-MacBook-Pro Documents % cd flink-quickstart-java

abdullah@Abdullahs-MacBook-Pro flink-quickstart-java % ls

pom.xml src target

abdullah@Abdullahs-MacBook-Pro flink-quickstart-java % mvn clean package

[INFO] Scanning for projects...

[INFO] 

[INFO] ---------------< org.apache.flink:flink-quickstart-java >---------------

[INFO] Building Flink Quickstart Job 0.0.1-SNAPSHOT

[INFO] --------------------------------[ jar ]---------------------------------

[INFO] 

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink-quickstart-java ---

[INFO] Deleting /Users/abdullah/Documents/flink-quickstart-java/target

[INFO] 

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink-quickstart-java ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 1 resource

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink-quickstart-java ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 2 source files to /Users/abdullah//Documents/flink-quickstart-java/target/classes

[INFO] 

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink-quickstart-java ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/abdullah/Documents/flink-quickstart-java/src/test/resources

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink-quickstart-java ---

[INFO] No sources to compile

[INFO] 

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ flink-quickstart-java ---

[INFO] No tests to run.

[INFO] 

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink-quickstart-java ---

[INFO] Building jar: /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar

[INFO] 

[INFO] --- maven-shade-plugin:3.1.1:shade (default) @ flink-quickstart-java ---

[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.15 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-slf4j-impl:jar:2.12.1 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-api:jar:2.12.1 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-core:jar:2.12.1 from the shaded jar.

[INFO] Replacing original artifact with shaded artifact.

[INFO] Replacing /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar with /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  3.811 s

[INFO] Finished at: 2021-03-04T13:50:25-06:00

[INFO] ------------------------------------------------------------------------


###### [2] Run the code for getting the workflow in the apache flink UI########## (it shows this  exception)


abdullah@Abdullahs-MacBook-Pro build-target % ./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar


Could not get job jar and dependencies from JAR file: JAR file does not exist: /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar


Use the help option (-h or --help) to get help on the command.


abdullah@Abdullahs-MacBook-Pro build-target % ./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar


------------------------------------------------------------

 The program finished with the following exception:


org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No operators defined in streaming topology. Cannot execute.

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)

at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)

at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)

at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)

at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)

at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)

Caused by: java.lang.IllegalStateException: No operators defined in streaming topology. Cannot execute.

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraphGenerator(StreamExecutionEnvironment.java:2024)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2015)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2000)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1839)

at org.apache.flink.flink_quickstart_java.StreamingJob.main(StreamingJob.java:62)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:564)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)

... 8 more










Screen Shot 2021-03-04 at 1.43.36 PM.png (190K) Download Attachment
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Running StreamingJob.java in apache flink workflow

Chesnay Schepler
The quickstarts provide you with a skeleton for building applications.
It provides you with pre-defined entrypoints (like StreamingJob) and the recommended maven setup to package this application into a jar.

However, they do not provide a running application. When you look at the StreamingJob class you will see that it contains very little; of note it does not contain any sources, functions or sinks. When you submitted this to Flink it was noticed that your application essentially doesn't contain anything, and it threw the error you saw.
The StreamingJob class contains some documentation in form of a comment that should guide you a bit further.

On 3/4/2021 9:01 PM, Abdullah bin Omar wrote:
Hi,

I have just created a maven project in eclipse IDE by using a  sample code (named by StramingJob.java) that is under flink-quickstart-java (this file is one of the source files on flink). See the attached pic. 

According to quickstarts, I used the "mvn clean package" command and got a jar file [1] 

So, the next step is to run [2] in flink UI by using the command:
./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar
Question:
However, I have got some exceptions. This is the reason not to connect the jar file in the apache flink UI.
What is the solution in this case?
###########[1] Building jar from terminal ####

abdullah@Abdullahs-MacBook-Pro Documents % cd flink-quickstart-java

abdullah@Abdullahs-MacBook-Pro flink-quickstart-java % ls

pom.xml src target

abdullah@Abdullahs-MacBook-Pro flink-quickstart-java % mvn clean package

[INFO] Scanning for projects...

[INFO] 

[INFO] ---------------< org.apache.flink:flink-quickstart-java >---------------

[INFO] Building Flink Quickstart Job 0.0.1-SNAPSHOT

[INFO] --------------------------------[ jar ]---------------------------------

[INFO] 

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink-quickstart-java ---

[INFO] Deleting /Users/abdullah/Documents/flink-quickstart-java/target

[INFO] 

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink-quickstart-java ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 1 resource

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink-quickstart-java ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 2 source files to /Users/abdullah//Documents/flink-quickstart-java/target/classes

[INFO] 

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink-quickstart-java ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/abdullah/Documents/flink-quickstart-java/src/test/resources

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink-quickstart-java ---

[INFO] No sources to compile

[INFO] 

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ flink-quickstart-java ---

[INFO] No tests to run.

[INFO] 

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink-quickstart-java ---

[INFO] Building jar: /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar

[INFO] 

[INFO] --- maven-shade-plugin:3.1.1:shade (default) @ flink-quickstart-java ---

[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.15 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-slf4j-impl:jar:2.12.1 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-api:jar:2.12.1 from the shaded jar.

[INFO] Excluding org.apache.logging.log4j:log4j-core:jar:2.12.1 from the shaded jar.

[INFO] Replacing original artifact with shaded artifact.

[INFO] Replacing /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar with /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  3.811 s

[INFO] Finished at: 2021-03-04T13:50:25-06:00

[INFO] ------------------------------------------------------------------------


###### [2] Run the code for getting the workflow in the apache flink UI########## (it shows this  exception)


abdullah@Abdullahs-MacBook-Pro build-target % ./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar

Could not get job jar and dependencies from JAR file: JAR file does not exist: /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT-shaded.jar


Use the help option (-h or --help) to get help on the command.

abdullah@Abdullahs-MacBook-Pro build-target % ./bin/flink run /Users/abdullah/Documents/flink-quickstart-java/target/flink-quickstart-java-0.0.1-SNAPSHOT.jar


------------------------------------------------------------

 The program finished with the following exception:


org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No operators defined in streaming topology. Cannot execute.

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)

at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)

at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)

at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)

at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)

at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)

Caused by: java.lang.IllegalStateException: No operators defined in streaming topology. Cannot execute.

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraphGenerator(StreamExecutionEnvironment.java:2024)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2015)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2000)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1839)

at org.apache.flink.flink_quickstart_java.StreamingJob.main(StreamingJob.java:62)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:564)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)

... 8 more
Free forum by Nabble Edit this page