Hey, I am new to flink and I have a question and want to see if anyone can help here.
So we have a s3 path that flink is monitoring that path to see new files available.
val avroInputStream_activity = env.readFile(format, path, FileProcessingMode.PROCESS_CONTINUOUSLY, 10000)
I am doing both internal and external check pointing and let's say there is a bad file came to the path and flink will do several retries. I want to take those bad files and let the process continue. However, since the file path persist in the checkpoint, when I try to resume from external checkpoint, it threw the following error on no file been found.
java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No such file or directory: s3a://myfile
Is there a way to skip this bad file and move on?
Thanks in advance.
Best,
Chengzhi Zhao