S3 file source parallelism reverting to 1
Posted by
Steve Whelan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/S3-file-source-parallelism-reverting-to-1-tp37311.html
Hi,
I have an S3 file source being consumed as FileProcessingMode.PROCESS_CONTINUOUSLY with a parallelism of 3. I can confirm the parallelism is set by printing it out. However, in the UI, the file source has a parallelism of 1. I'm not changing it after its being initially set.
DataStream s = env.readFile(
new JsonInputFormat(...),
filePath,
FileProcessingMode.PROCESS_CONTINUOUSLY,
5,
myTypeInformation)
.setParallelism(3);
System.out.println(s.getParallelism()); // prints 3
The DataStreamSource is a parallel operator otherwise `setParallelism(3)` would throw an `IllegalArgumentException`[1]. The only other thing I do with the DataStream is register it with the TableEnvironment.
tableEnvironment.registerDataStream("my_table", dataStream);
Is Flink resetting it to 1 for some reason? I'm running v1.9.0.
Thanks,
Steve