Hello,
I’m running a single Flink Job Manager with a Task Manager in Docker containers with Java 8. They are remotely located (
flink.example.com).
I’m submitting a job from my desktop and passing the job to the Job Manager with -m
flink.example.com:6123, which seems to work well. I’m doing a search on an S3 system located at
s3.example.com.
The problem is that in order to have access to the S3 system, the $HADOOP_CONFIG/core-site.xml and $FLINK_HOME/flink-conf.yaml need to be configured for it at the Job Manager and Task Manager level, which means they are tied to that particular endpoint (including my access key and secret key). Is there someway I can specify the configuration only in my application so I can leave my Flink server cluster to be mostly generic?
Thank you!
Dustin
Locked
