RestClusterClient locks file after calling `submitJob(JobGraph)` method on Windows OS

[Vladislav Keda]

Hi Flink Community,

I was trying to submit a flink job on a standalone cluster using RestClusterClient. After waiting for job submission, I got JobID correctly and tried to delete the source jar file. But then I got the exception:


java.nio.file.FileSystemException: /path/to/jar: Процесс не может получить доступ к файлу, так как этот файл занят другим процессом.

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at ru.glowbyte.streaming.manager.TestApp.main(TestApp.java:57)


See the program code below:

    public static void main(String[] args) {
        final Configuration flinkConf = new Configuration();
        flinkConf.set(RestOptions.ADDRESS, "localhost");
        flinkConf.set(RestOptions.PORT, 8081);

        final File jarFile = new File("/path/to/jar");

        try {
            final RestClusterClient<StandaloneClusterId> client = new RestClusterClient<>(flinkConf, StandaloneClusterId.getInstance());

            final PackagedProgram packagedProgram = PackagedProgram.newBuilder()
                    .setJarFile(jarFile)
                    .setConfiguration(flinkConf)
                    .build();

            final JobGraph jobGraph = PackagedProgramUtils.createJobGraph(
                    packagedProgram,
                    flinkConf,
                    1,
                    true);

            final DetachedJobExecutionResult jobExecutionResult = client.submitJob(jobGraph)
                    .thenApply(DetachedJobExecutionResult::new)
                    .get();

            System.out.println(jobExecutionResult.getJobID());
        } catch (Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }

        try {
            // failed to delete jar on Windows OS, process cannot access the file
            Files.delete(jarFile.toPath());
        } catch (IOException ex) {
            ex.printStackTrace();
            System.exit(1);
        }
    }

I execute this code on Windows OS. I think that after calling the `submitJob(JobGraph)` method, the jar file remains unclosed.

Is it a bug or not? Maybe I'm doing something wrong?

[Chesnay Schepler]

This is a known issue: https://issues.apache.org/jira/browse/FLINK-9844

On 9/11/2020 8:34 PM, Vladislav Keda wrote:

Hi Flink Community,

I was trying to submit a flink job on a standalone cluster using RestClusterClient. After waiting for job submission, I got JobID correctly and tried to delete the source jar file. But then I got the exception:


java.nio.file.FileSystemException: /path/to/jar: Процесс не может получить доступ к файлу, так как этот файл занят другим процессом.

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at ru.glowbyte.streaming.manager.TestApp.main(TestApp.java:57)


See the program code below:

    public static void main(String[] args) {
        final Configuration flinkConf = new Configuration();
        flinkConf.set(RestOptions.ADDRESS, "localhost");
        flinkConf.set(RestOptions.PORT, 8081);

        final File jarFile = new File("/path/to/jar");

        try {
            final RestClusterClient<StandaloneClusterId> client = new RestClusterClient<>(flinkConf, StandaloneClusterId.getInstance());

            final PackagedProgram packagedProgram = PackagedProgram.newBuilder()
                    .setJarFile(jarFile)
                    .setConfiguration(flinkConf)
                    .build();

            final JobGraph jobGraph = PackagedProgramUtils.createJobGraph(
                    packagedProgram,
                    flinkConf,
                    1,
                    true);

            final DetachedJobExecutionResult jobExecutionResult = client.submitJob(jobGraph)
                    .thenApply(DetachedJobExecutionResult::new)
                    .get();

            System.out.println(jobExecutionResult.getJobID());
        } catch (Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }

        try {
            // failed to delete jar on Windows OS, process cannot access the file
            Files.delete(jarFile.toPath());
        } catch (IOException ex) {
            ex.printStackTrace();
            System.exit(1);
        }
    }

I execute this code on Windows OS. I think that after calling the `submitJob(JobGraph)` method, the jar file remains unclosed.

Is it a bug or not? Maybe I'm doing something wrong?

[Vladislav Keda]

Thanks, I missed the fact that this issue was created early.

вс, 13 сент. 2020 г., 14:58 Chesnay Schepler <[hidden email]>:

This is a known issue: https://issues.apache.org/jira/browse/FLINK-9844

On 9/11/2020 8:34 PM, Vladislav Keda wrote:

Hi Flink Community,

I was trying to submit a flink job on a standalone cluster using RestClusterClient. After waiting for job submission, I got JobID correctly and tried to delete the source jar file. But then I got the exception:


java.nio.file.FileSystemException: /path/to/jar: Процесс не может получить доступ к файлу, так как этот файл занят другим процессом.

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at ru.glowbyte.streaming.manager.TestApp.main(TestApp.java:57)


See the program code below:

    public static void main(String[] args) {
        final Configuration flinkConf = new Configuration();
        flinkConf.set(RestOptions.ADDRESS, "localhost");
        flinkConf.set(RestOptions.PORT, 8081);

        final File jarFile = new File("/path/to/jar");

        try {
            final RestClusterClient<StandaloneClusterId> client = new RestClusterClient<>(flinkConf, StandaloneClusterId.getInstance());

            final PackagedProgram packagedProgram = PackagedProgram.newBuilder()
                    .setJarFile(jarFile)
                    .setConfiguration(flinkConf)
                    .build();

            final JobGraph jobGraph = PackagedProgramUtils.createJobGraph(
                    packagedProgram,
                    flinkConf,
                    1,
                    true);

            final DetachedJobExecutionResult jobExecutionResult = client.submitJob(jobGraph)
                    .thenApply(DetachedJobExecutionResult::new)
                    .get();

            System.out.println(jobExecutionResult.getJobID());
        } catch (Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }

        try {
            // failed to delete jar on Windows OS, process cannot access the file
            Files.delete(jarFile.toPath());
        } catch (IOException ex) {
            ex.printStackTrace();
            System.exit(1);
        }
    }

I execute this code on Windows OS. I think that after calling the `submitJob(JobGraph)` method, the jar file remains unclosed.

Is it a bug or not? Maybe I'm doing something wrong?