[Flink 1.10] Classpath doesn't include custom files in lib/

[Maxim Parkachov]

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank

[Yang Wang]

Hi Maxim Parkachov,

The users files also have been shipped to JobManager and TaskManager. However, it

is not directly added to the classpath. Instead, the parent directory is added to the

classpath. This changes are to make resource classloading work. You could check more

information here[1].

[1]. https://issues.apache.org/jira/browse/FLINK-13127

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月15日周六 上午12:58写道：

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank

[Maxim Parkachov]

Hi Yang,

thanks, this explains why classpath behavior changed, but now I struggle to

understand how I could overwrite resource, which is already shipped in job jar.

Before I had job.properties files in JAR in under resources/lib/job.properties

for local development and deploying on cluster it was overwritten 

with environment specific settings in  lib/job.properties of flink distribution. 

Now this doesn't seem to work.  I'm using: 

getClass.getClassLoader.getResource("lib/job.properties")

to get file. Could it be the problem ?

Thanks,
Maxim.

On Mon, Feb 17, 2020 at 4:12 AM Yang Wang <[hidden email]> wrote:

Hi Maxim Parkachov,

The users files also have been shipped to JobManager and TaskManager. However, it

is not directly added to the classpath. Instead, the parent directory is added to the

classpath. This changes are to make resource classloading work. You could check more

information here[1].

[1]. https://issues.apache.org/jira/browse/FLINK-13127

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月15日周六 上午12:58写道：

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank

[Yang Wang]

Hi Maxim,

I have verified that the following two ways could both work.

getClass().getClassLoader().getResource("lib/job.properties")

getClass().getClassLoader().getResource("job.properties") 

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月17日周一 下午6:47写道：

Hi Yang,

thanks, this explains why classpath behavior changed, but now I struggle to

understand how I could overwrite resource, which is already shipped in job jar.

Before I had job.properties files in JAR in under resources/lib/job.properties

for local development and deploying on cluster it was overwritten 

with environment specific settings in  lib/job.properties of flink distribution. 

Now this doesn't seem to work.  I'm using: 

getClass.getClassLoader.getResource("lib/job.properties")

to get file. Could it be the problem ?

Thanks,
Maxim.

On Mon, Feb 17, 2020 at 4:12 AM Yang Wang <[hidden email]> wrote:

Hi Maxim Parkachov,

The users files also have been shipped to JobManager and TaskManager. However, it

is not directly added to the classpath. Instead, the parent directory is added to the

classpath. This changes are to make resource classloading work. You could check more

information here[1].

[1]. https://issues.apache.org/jira/browse/FLINK-13127

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月15日周六 上午12:58写道：

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank

[Maxim Parkachov]

Hi Yang,

I've just tried your suggestions, but, unfortunately, in yarn per job mode it doesn't work, both commands return null.

I double checked that file is shipped to yarn container, but I feel that it happens later in process.

At the moment I'm reading file with File interface, instead of getting it as resource, which I do in local mode.

Regards,

Maxim.

On Mon, Feb 17, 2020 at 3:03 PM Yang Wang <[hidden email]> wrote:

Hi Maxim,

I have verified that the following two ways could both work.

getClass().getClassLoader().getResource("lib/job.properties")

getClass().getClassLoader().getResource("job.properties") 

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月17日周一 下午6:47写道：

Hi Yang,

thanks, this explains why classpath behavior changed, but now I struggle to

understand how I could overwrite resource, which is already shipped in job jar.

Before I had job.properties files in JAR in under resources/lib/job.properties

for local development and deploying on cluster it was overwritten 

with environment specific settings in  lib/job.properties of flink distribution. 

Now this doesn't seem to work.  I'm using: 

getClass.getClassLoader.getResource("lib/job.properties")

to get file. Could it be the problem ?

Thanks,
Maxim.

On Mon, Feb 17, 2020 at 4:12 AM Yang Wang <[hidden email]> wrote:

Hi Maxim Parkachov,

The users files also have been shipped to JobManager and TaskManager. However, it

is not directly added to the classpath. Instead, the parent directory is added to the

classpath. This changes are to make resource classloading work. You could check more

information here[1].

[1]. https://issues.apache.org/jira/browse/FLINK-13127

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月15日周六 上午12:58写道：

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank

[Yang Wang]

Hi Maxim,

Both Yarn per-job and session cluster should work. Since before the JobManager and TaskManager

launcher, Yarn NodeManager could guarantee that all the local resources have been localized and

accessible.

If do not want to use the getResource to read the file and use File interface instead, you should use

"lib/job.properties" as your file path. Because the PWD of JobManager and TaskManager process is

the Yarn container workdir. 

If it still could not work, do you mind to share your codes and TaskManager logs?

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月18日周二 上午12:31写道：

Hi Yang,

I've just tried your suggestions, but, unfortunately, in yarn per job mode it doesn't work, both commands return null.

I double checked that file is shipped to yarn container, but I feel that it happens later in process.

At the moment I'm reading file with File interface, instead of getting it as resource, which I do in local mode.

Regards,

Maxim.

On Mon, Feb 17, 2020 at 3:03 PM Yang Wang <[hidden email]> wrote:

Hi Maxim,

I have verified that the following two ways could both work.

getClass().getClassLoader().getResource("lib/job.properties")

getClass().getClassLoader().getResource("job.properties") 

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月17日周一 下午6:47写道：

Hi Yang,

thanks, this explains why classpath behavior changed, but now I struggle to

understand how I could overwrite resource, which is already shipped in job jar.

Before I had job.properties files in JAR in under resources/lib/job.properties

for local development and deploying on cluster it was overwritten 

with environment specific settings in  lib/job.properties of flink distribution. 

Now this doesn't seem to work.  I'm using: 

getClass.getClassLoader.getResource("lib/job.properties")

to get file. Could it be the problem ?

Thanks,
Maxim.

On Mon, Feb 17, 2020 at 4:12 AM Yang Wang <[hidden email]> wrote:

Hi Maxim Parkachov,

The users files also have been shipped to JobManager and TaskManager. However, it

is not directly added to the classpath. Instead, the parent directory is added to the

classpath. This changes are to make resource classloading work. You could check more

information here[1].

[1]. https://issues.apache.org/jira/browse/FLINK-13127

Best,

Yang

Maxim Parkachov <[hidden email]> 于2020年2月15日周六 上午12:58写道：

Hi everyone,

I'm trying to run my job with flink 1.10 with YARN cluster per-job mode. In the previous versions all files in lib/ folder were automatically included in classpath. Now, with 1.10 I see only *.jar files are included in classpath. but not "other" files. Is this deliberate change or bug ?

Generally, what is recommended way to include custom files in classpath and ship it during start to all containers ?

Thank