Test FileUtilsTest.testDeleteDirectory failed when building Flink

[Paul Lam]

Hi all,

Recently we migrate Flink build to a new docker image, after which the build job always fails with test errors on local file system permissions. 

For example: FileUtilsTest.testDeleteDirectory:129 this should fail with an exception.

I notice the following statements in the javadoc of `java.io.File.setWritable`:

> On some platforms it may be possible to start the Java virtual machine with special privileges that allow it to modify files that disallow write operations.

I think it’s what the test is designed for and where the problem lies. 

Could anyone help me with this? Thanks a lot!

WRT the environment:

- Flink version: 1.7.1

- JDK: open jdk 1.8.0_111

- OS version: debian 8

Best,

Paul Lam

[Fabian Hueske-2]

Hi Paul,

Which components (Flink, JDK, Docker base image, ...) are you upgrading and which versions do you come from?

I think it would be good to check how (and with which options) the JVM in the container is started.

Best, Fabian

Am Fr., 15. Feb. 2019 um 09:50 Uhr schrieb Paul Lam <[hidden email]>:

Hi all,

Recently we migrate Flink build to a new docker image, after which the build job always fails with test errors on local file system permissions. 

For example: FileUtilsTest.testDeleteDirectory:129 this should fail with an exception.

I notice the following statements in the javadoc of `java.io.File.setWritable`:

> On some platforms it may be possible to start the Java virtual machine with special privileges that allow it to modify files that disallow write operations.

I think it’s what the test is designed for and where the problem lies. 

Could anyone help me with this? Thanks a lot!

WRT the environment:

- Flink version: 1.7.1

- JDK: open jdk 1.8.0_111

- OS version: debian 8

Best,

Paul Lam

[Chesnay Schepler]

I ran into a similar issue when i looked at other CI solutions; imo we shouldn't rely on the result of setWritable but instead actually verify whether the forbidden operation (i.e. creating/writing files) throws an error.

Back then also created a JIRA: https://issues.apache.org/jira/browse/FLINK-5970

On 18.02.2019 14:10, Fabian Hueske wrote:

Hi Paul,

Which components (Flink, JDK, Docker base image, ...) are you upgrading and which versions do you come from?

I think it would be good to check how (and with which options) the JVM in the container is started.

Best, Fabian

Am Fr., 15. Feb. 2019 um 09:50 Uhr schrieb Paul Lam <[hidden email]>:

Hi all,

Recently we migrate Flink build to a new docker image, after which the build job always fails with test errors on local file system permissions. 

For example: FileUtilsTest.testDeleteDirectory:129 this should fail with an exception.

I notice the following statements in the javadoc of `java.io.File.setWritable`:

> On some platforms it may be possible to start the Java virtual machine with special privileges that allow it to modify files that disallow write operations.

I think it’s what the test is designed for and where the problem lies. 

Could anyone help me with this? Thanks a lot!

WRT the environment:

- Flink version: 1.7.1

- JDK: open jdk 1.8.0_111

- OS version: debian 8

Best,

Paul Lam

[Paul Lam]

Hi Chesnay & Fabian,

Thanks for your replies.

I found it should be related to the CI runner. I moved to gitlab CI which runs the script as root user by default, so it is always able to remove a write protected file.

Best,

Paul Lam

在 2019年2月20日，17:08，Chesnay Schepler <[hidden email]> 写道：

I ran into a similar issue when i looked at other CI solutions; imo we shouldn't rely on the result of setWritable but instead actually verify whether the forbidden operation (i.e. creating/writing files) throws an error.

Back then also created a JIRA: https://issues.apache.org/jira/browse/FLINK-5970

On 18.02.2019 14:10, Fabian Hueske wrote:

Hi Paul,

Which components (Flink, JDK, Docker base image, ...) are you upgrading and which versions do you come from?

I think it would be good to check how (and with which options) the JVM in the container is started.

Best, Fabian

Am Fr., 15. Feb. 2019 um 09:50 Uhr schrieb Paul Lam <[hidden email]>:

Hi all,

Recently we migrate Flink build to a new docker image, after which the build job always fails with test errors on local file system permissions. 

For example: FileUtilsTest.testDeleteDirectory:129 this should fail with an exception.

I notice the following statements in the javadoc of `java.io.File.setWritable`:

> On some platforms it may be possible to start the Java virtual machine with special privileges that allow it to modify files that disallow write operations.

I think it’s what the test is designed for and where the problem lies. 

Could anyone help me with this? Thanks a lot!

WRT the environment:

- Flink version: 1.7.1

- JDK: open jdk 1.8.0_111

- OS version: debian 8

Best,

Paul Lam