wait for "writeAsText" to finish

[Lydia Ickler]

Hi,

I have a program that contains a preprocessing with Flink Objects and at the end writes the result with „result.writeAsText(„...“)“.
After that I call a method that is basically a MapReduce-Job (actually only a Map-Job) which depends on the written file.

So what is the smartest way to delay the execution of the Map-Job until the file is written completely?
Right now I I am doing it the following way:

val written = result.writeAsText(„…“)
if(written.getDataSet.count() > 0){ ...do Map-Job...}

Thanks in advance!
Best regards,
Lydia

[Chesnay Schepler]

Hello,

Since ExecutionEnvironment#execute() blocks until the job is finished you should be able to just do this:

data.writeAsText();
env.execute();
{ do Map-Job }

Note that your current solution is wrong, as it translates to this:

DataSet result = ...
result.writeAsText();
if (result.count() > 0) { ... do Map-Job ... }

Regards,
Chesnay

On 20.04.2017 09:46, Lydia wrote:

Hi, 

I have a program that contains a preprocessing with Flink Objects and at the end writes the result with „result.writeAsText(„...“)“.
After that I call a method that is basically a MapReduce-Job (actually only a Map-Job) which depends on the written file.

So what is the smartest way to delay the execution of the Map-Job until the file is written completely?
Right now I I am doing it the following way:

val written = result.writeAsText(„…“)
if(written.getDataSet.count() > 0){ ...do Map-Job...}

Thanks in advance!
Best regards,
Lydia