External classpath

[Vishwas Siravara]

Hi guys,

I m very close to deploying my application in production so I am trying to externalize some of the config files which has to be available on the classpath when I run my application via flink command line interface. 

From the flink doc I can add to class path by 

-C,--classpath <url>                 Adds a URL to each user code
                                          classloader  on all nodes in the
                                          cluster. The paths must specify a
                                          protocol (e.g. file://) and be
                                          accessible on all nodes (e.g. by means
                                          of a NFS share). You can use this
                                          option multiple times for specifying
                                          more than one URL. The protocol must
                                          be supported by the {@link
                                          java.net.URLClassLoader}.

So my job has 8 task managers( 8 different nodes) with 8 slots in each . Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ? For instance on Node 1 the config files are in  the directory is /home/was/classpathconfig/  and the same on every node. Does it have to be on an NFS ? 

My command looks like this flink run -C file://home/was/classpathconfig/ -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Also I tried to put my files in s3 and tried to run flink run -C s3://flink.dev/config -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Bad syntax for classpath: s3://flink.dev/config

s3 does support URLClassLoader but I get the error saying bad syntax. 

Please let me know your thoughts. Thanks a lot to this community , I was able to write my code in a week. 

Thanks,

Vishwas 

[Biao Liu]

Hi Vishwas,

> Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ?

It can be local files on each machine.

> -C file://home/was/classpathconfig/

I guess the problem is your URI format of local file is not correct. Try to use "-C file:///home/was/classpathconfig/" instead.

BTW, if it's a directory, it must end with '/' [1].

> Also I tried to put my files in s3 and tried to run flink run -C s3://flink.dev/config -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Currently it doesn't support s3 or other distributed file system as the parameter of -C.

1. https://docs.oracle.com/javase/8/docs/api/java/net/URLClassLoader.html

On Thu, 15 Aug 2019 at 07:51, Vishwas Siravara <[hidden email]> wrote:

Hi guys,

I m very close to deploying my application in production so I am trying to externalize some of the config files which has to be available on the classpath when I run my application via flink command line interface. 

From the flink doc I can add to class path by 

-C,--classpath <url>                 Adds a URL to each user code
                                          classloader  on all nodes in the
                                          cluster. The paths must specify a
                                          protocol (e.g. file://) and be
                                          accessible on all nodes (e.g. by means
                                          of a NFS share). You can use this
                                          option multiple times for specifying
                                          more than one URL. The protocol must
                                          be supported by the {@link
                                          java.net.URLClassLoader}.

So my job has 8 task managers( 8 different nodes) with 8 slots in each . Does my external class path have to be on NFS share ? Can I not have the config directory on each machine in the same location ? For instance on Node 1 the config files are in  the directory is /home/was/classpathconfig/  and the same on every node. Does it have to be on an NFS ? 

My command looks like this flink run -C file://home/was/classpathconfig/ -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Also I tried to put my files in s3 and tried to run flink run -C s3://flink.dev/config -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid

Bad syntax for classpath: s3://flink.dev/config

s3 does support URLClassLoader but I get the error saying bad syntax. 

Please let me know your thoughts. Thanks a lot to this community , I was able to write my code in a week. 

Thanks,

Vishwas