Re: Monitoring folder in flink

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Monitoring-folder-in-flink-tp16327p16385.html

Hi,

with PROCESS_CONTINUOUSLY the application monitors the directory and processes new arriving files or files that have been modified. In this case the application never terminates because it is waiting for new files to appear.

With PROCESS_ONCE, the content of a directory is processed as it was when the application was started. Once all files are processed the application terminates.

What kind of behavior are you looking for?

Best, Fabian

2017-10-22 9:37 GMT+02:00 Sugandha Amatya <[hidden email]>:

I have folder where new files arrive at schedule. Why is my flink readfile not reading new files. I have used but *PROCESS_ONCE* and *PROCESS_CONTINUOUSLY*. When I use *PROCESS_CONTINUOUSLY* it reads the same file but the execution does not terminate whereas for PROCESS_ONCE it terminates in IDE.

    String path = "C:\\test";

    

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    

    TextInputFormat format = new TextInputFormat(new org.apache.flink.core.fs.Path(path));

    

    DataStream<String> inputStream = env.readFile(format, path, FileProcessingMode.PROCESS_ONCE, 100);

Here at stackoverflow.