What is the right way to add classpath?

[bupt_ljy]

Hi,all

  My program needs some dependencies before it’s submitted to yarn. Like:

  ```

  stream.filter(new FilterService()).print()

  env.execute()

  ```

  I use external dependency in FilterService, and the program reports NoClassDefFoundError at     

  org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)

  Now my solution is changing the bin/flink and add my classpath into it.

  Any better ideas?

  Thanks.

  Best, Jiayi Liao

[Yun Tang]

Hi Jiayi

As far as I know, there exist three ways:

1. Build the fat-application jar with dependencies using maven-shade-plugin or maven-assembly-plugin.
2. Copy the dependency jars to local ${FLINK_HOME}/lib  folder.
3. Submit the job with -yt,--yarnship <args> command, please refer to https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/cli.html#usage

Best

Yun Tang

---

From: bupt_ljy <[hidden email]>
Sent: Wednesday, September 12, 2018 16:34
To: user
Subject: What is the right way to add classpath?

 

Hi,all

  My program needs some dependencies before it’s submitted to yarn. Like:

  ```

  stream.filter(new FilterService()).print()

  env.execute()

  ```

  I use external dependency in FilterService, and the program reports NoClassDefFoundError at     

  org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)

  Now my solution is changing the bin/flink and add my classpath into it.

  Any better ideas?

  Thanks.

  Best, Jiayi Liao

[bupt_ljy]

Hi,Yun Tang

   Thanks for help. The first option makes the package process heavy, the second will make a change to flink’s lib folder. And the -yt cannot help also, because I need these dependencies before it’s submitted on yarn, and I did use -yt to submit my job and failed.

  Best, Jiayi Liao

 Original Message 

Sender: Yun Tang<[hidden email]>

Recipient: bupt_ljy<[hidden email]>; user<[hidden email]>

Date: Wednesday, Sep 12, 2018 17:53

Subject: Re: What is the right way to add classpath?

Hi Jiayi

As far as I know, there exist three ways:

1. Build the fat-application jar with dependencies using maven-shade-plugin or maven-assembly-plugin.
2. Copy the dependency jars to local ${FLINK_HOME}/lib  folder.
3. Submit the job with -yt,--yarnship <args> command, please refer to https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/cli.html#usage

Best

Yun Tang

---

From: bupt_ljy <[hidden email]>
Sent: Wednesday, September 12, 2018 16:34
To: user
Subject: What is the right way to add classpath?

 

Hi,all

  My program needs some dependencies before it’s submitted to yarn. Like:

  ```

  stream.filter(new FilterService()).print()

  env.execute()

  ```

  I use external dependency in FilterService, and the program reports NoClassDefFoundError at     

  org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)

  Now my solution is changing the bin/flink and add my classpath into it.

  Any better ideas?

  Thanks.

  Best, Jiayi Liao