Re: Checking for existance of output directory/files before running a batch job

Posted by rmetzger0 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Checking-for-existance-of-output-directory-files-before-running-a-batch-job-tp8573p8583.html

Hi Niels,

I assume the directoryName you are passing doesn't have the file system prefix (hdfs:// or s3://, ...) specified.

In those cases, Path.getFileSystem() is looking up the default file system prefix from the configuration.

Probably the environment where you are submitting the job from doesn't have the correct Flink configuration.

How are you starting the Flink job? Does the conf/flink-conf.yaml have the correct hdfs configuration?

Regards,

Robert

On Thu, Aug 18, 2016 at 2:02 PM, Niels Basjes <[hidden email]> wrote:

Hi,

I have a batch job that I run on yarn that creates files in HDFS.

I want to avoid running this job at all if the output already exists.

So in my code (before submitting the job into yarn-session) I do this:

    String directory = "foo";

    Path directory = new Path(directoryName);
    FileSystem fs = directory.getFileSystem();

    if (!fs.exists(directory)) {

        // run the job

    }

What I found is that this code apparently checks the 'wrong' file system. (I always get 'false' even if it exists in hdfs)

I checked the API of the execution environment yet I was unable to get the 'correct' filesystem from there.

What is the proper way to check this?

--