Failure running Flink locally with flink-s3-fs-hadoop + AWS SDK v2 as a dependency
I'm trying to run Flink with AWS SDK v2 and restore a Flink savepoint that's stored in S3.
When adding the dependency on flink-s3-fs-hadoop locally, I fail to run Flink at runtime with the following error:
Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type interface org.apache.http.Header; not a subtype of implementation type interface org.apache.http.NameValuePair
This happens because the flink-s3 JAR is a fat JAR containing an old version of org.apache.http.HttpMessage which does not implement the NameValuePair interface (there was a non backwards compatible change in the library, see https://github.com/aws/aws-sdk-java-v2/issues/652)
Since the library itself isn't shaded, it gets loaded first overriding the HttpClient newer version I have on the classpath and blows up when bootstrapping Flink.