terminate and restart Flink app again, data may lose?

[Yu, Yucai]

Hi,

 

We are using StreamingFileSink in our Flink apps (1.7), we found when terminating and restarting app again, a lot of files will in “ingress” status, never get a chance to commit.

It may leads to lose data, any idea?

 

Thanks,

Lex

[Kostas Kloudas-2]

Hi Yucai,

How do you terminate the application? Do you take a savepoint before and you restart from that afterwards?

Also do you mean that the in-progress files do not get deleted after being committed or that they are not committed?

In the screenshot it seems that only part-0-3 is still not committed. 

The rest seem to be finalised.

I assume you are using HDFS, right?

Cheers,

Kostas

On Thu, Apr 2, 2020 at 3:11 PM Yu, Yucai <[hidden email]> wrote:

Hi,

 

We are using StreamingFileSink in our Flink apps (1.7), we found when terminating and restarting app again, a lot of files will in “ingress” status, never get a chance to commit.

It may leads to lose data, any idea?

 

Thanks,

Lex