Reading files from multiple subdirectories

[Lorenzo Nicora]

Hi,

related to the same case I am discussing in another thread, but not related to AVRO this time :) 

I need to ingest files a S3 Sink Kafka Connector periodically adds to an S3 bucket.

Files are bucketed by date time as it often happens.

Is there any way, using Flink only, to monitor a base-path and detect new files in any subdirectories? 

Or I need to use something external to move new files in a single directory?

I am currently using  

env.readFile(inputFormat, path, PROCESS_CONTINUOUSLY, 60000)

with AvroInputFormat, but it seems it can only monitor a single directory

Cheers

[Yun Gao]

Hi Lorenzo,

    Read from a previouse thread [1] and the source code, I think you may set inputFormat.setNestedFileEnumeration(true) to also scan the nested files.

Best,

Yun

[1] https://lists.apache.org/thread.html/86a23b4c44d92c3adeb9ff4a708365fe4099796fb32deb6319e0e17f%40%3Cuser.flink.apache.org%3E

------------------------------------------------------------------
Sender:Lorenzo Nicora<[hidden email]>
Date:2020/06/11 21:31:20
Recipient:user<[hidden email]>
Theme:Reading files from multiple subdirectories

Hi,

related to the same case I am discussing in another thread, but not related to AVRO this time :) 

I need to ingest files a S3 Sink Kafka Connector periodically adds to an S3 bucket.

Files are bucketed by date time as it often happens.

Is there any way, using Flink only, to monitor a base-path and detect new files in any subdirectories? 

Or I need to use something external to move new files in a single directory?

I am currently using  

env.readFile(inputFormat, path, PROCESS_CONTINUOUSLY, 60000)

with AvroInputFormat, but it seems it can only monitor a single directory

Cheers