(DISSCUSS) flink cli need load '--classpath' files
Locked
 
|
Hi all When I start a flinkcluster in session mode, It include jm/tm. And then I submit a job like ‘bin/flink run —jobmanager “ip:8081” —class path a.jar’. Even the a.jar in all jm/tm and ‘bin/flink’ mechine . It will throw exception “ /opt/flink/bin/flink run --jobmanager ip:8081 --class com.netease.java.TopSpeedWindowing --parallelism 1 --detached /opt/flink/job/kafkaDemo19-1.0-SNAPSHOT.jar --classpath file:///opt/flink/job/fastjson-1.2.66.jarStarting execution of program Executing TopSpeedWindowing example with default input data set. Use --input to specify file input. java.lang.NoClassDefFoundError: com/alibaba/fastjson/JSON at com.netease.java.TopSpeedWindowing.main(TopSpeedWindowing.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)” As I read the code , flink cli have not load the —classspath jar, So It seems a bug about the flink cli. Are you agree with me?
Best,
Ouywl
|
Re: (DISSCUSS) flink cli need load '--classpath' files
|
|
Hi,
first a preliminary question: does the jar file contain com.alibaba.fastjson.JSON? Could you maybe list the contents of the jar here? Best, Aljoscha On 06.03.20 13:25, ouywl wrote: > Hi all > When I start a flinkcluster in session mode, It include jm/tm. And then I > submit a job like ‘bin/flink run —jobmanager “ip:8081” —class path a.jar’. Even > the a.jar in all jm/tm and ‘bin/flink’ mechine . It will throw exception “ > /opt/flink/bin/flink run --jobmanager ip:8081 --class > com.netease.java.TopSpeedWindowing --parallelism 1 --detached > /opt/flink/job/kafkaDemo19-1.0-SNAPSHOT.jar --classpath > file:///opt/flink/job/fastjson-1.2.66.jar > Starting execution of program > Executing TopSpeedWindowing example with default input data set. > Use --input to specify file input. > java.lang.NoClassDefFoundError: com/alibaba/fastjson/JSON > at com.netease.java.TopSpeedWindowing.main(TopSpeedWindowing.java:98) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)” > As I read the code , flink cli have not load the —classspath jar, So It seems > a bug about the flink cli. Are you agree with me? > Best, > Ouywl > |
|
|
I think the problem is that --classpath should be before the user jar, i.e., /opt/flink/job/kafkaDemo19-1.0-SNAPSHOT.jar Best, tison. Aljoscha Krettek <[hidden email]> 于2020年3月6日周五 下午10:03写道： Hi, |
Re: (DISSCUSS) flink cli need load '--classpath' files
|
|
In reply to this post by Aljoscha Krettek
Hi ouywl, As I know, "--classpath" should be in front of jar file, it means: /opt/flink/bin/flink run --jobmanager ip:8081 --class com.netease.java.TopSpeedWindowing --parallelism 1 --detached --classpath file:///opt/flink/job/fastjson-1.2.66.jar /opt/flink/job/kafkaDemo19-1.0-SNAPSHOT.jar You can have a try. Best, Jingsong Lee On Fri, Mar 6, 2020 at 10:03 PM Aljoscha Krettek <[hidden email]> wrote: Hi, Best, Jingsong Lee |
|
|
In reply to this post by tison
It is because as implementation when we parse command line argument it "stopAtNonOptions" at the arbitrary content user jar. All arguments later will be regarded as args passed to user main. For user serving, when you run `./bin/flink run -h`, it prints Action "run" compiles and runs a program. Syntax: run [OPTIONS] <jar-file> <arguments> that explicit explains the format. Best, tison. tison <[hidden email]> 于2020年3月6日周五 下午10:22写道：
|
Re: (DISSCUSS) flink cli need load '--classpath' files
|
|
In reply to this post by tison
Hi tison and Aljoscha, Do you think "--classpath can not be in front of jar file" is an improvement? Or need documentation? Because I used to be confused. Best, Jingsong Lee On Fri, Mar 6, 2020 at 10:22 PM tison <[hidden email]> wrote:
Best, Jingsong Lee |
|
|
I think tison's answer is on point. All the Flink cli options should be specified before the user jar. We have a very clear help message. Syntax: run [OPTIONS] <jar-file> <arguments> Best, Yang Jingsong Li <[hidden email]> 于2020年3月6日周五 下午10:27写道：
|
|
|
In reply to this post by Jingsong Li
Hi Jingsong, I think your propose is "--classpath can occur behind the jar file". Generally speaking I agree on that it is a painful required format that users tend to just ignore that order how an option occurs. So it is +1 from my side to loose the constraint. However, for the migration and implementation part, things go into a bit tricky. For user interface, let's say we only enable --classpath to occur behind the jar file, at least the semantic changes if there is a user pass --classpath intended to be a main argument. Besides, said we fix on the library commons-cli to implement the CLI, it would be a bit tricky we implement such special taken logic. Accidentally I encounter similar CLI problem recently so here are some of my thoughts about the problem, 1. I agree that for options, users tend to treat <jar-file> the same as [OPTIONS] and mix up the order. It would be an improvement we loss the constraint. 2. Then, we still have to introduce something that users specify their args for the main method. 3. In order to achieve 2, there is a mature solution in shell scripting that use double-dash(--) to to signify the end of command options. 4. Now, if we keep <jar-file> as position argument, to support mix-ordered position argument & named argument, we might switch to other library such as argparse4j since commons-cli doesn't support position argument. An alternative is we change <jar-file> as named argument but then we double break user interface. Though, it will break user interface so we firstly **MUST** start a discussion and see whether the community think of it and if so, how to integrate it. For me, read the doc is an easy solution to save us from breaking user interface. I don't stick to loose the constraint. Best, tison. Jingsong Li <[hidden email]> 于2020年3月6日周五 下午10:27写道：
|
|
|
I tend to not change the current behavior. For other framework(e.g. hadoop, yarn), the arguments after user jar are also parsed by user `main()`, not the framework client. Best, Yang tison <[hidden email]> 于2020年3月6日周五 下午10:55写道：
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |