I need to set up a dockerized session cluster using Flink 1.8.2 for development and troubleshooting. We are bound to 1.8.2 as we are deploying to AWS Kinesis Data Analytics for Flink.
I am using an image based on the semi-official flink:1.8-scala_2.11
I need to add to my dockerized cluster support for S3 Hadoop File System (s3a://), we have on KDA out of the box.
Note I do not want to add dependencies to the job directly, as I want to deploy locally exactly the same JAR I deploy to KDA.
Flink 1.8 docs [1] say is supported out of the box but does not look to be the case for dockerised version.
I am getting "Could not find a file system implementation for scheme 's3a'" and "Hadoop is not in the classpath/dependencies".
I assume I need to create a customised docker image, extending flink:1.8-scala_2.11, but I do not understand how to add support for S3 Hadoop FS.
Can someone please point me in the right direction? Docs or examples?