Re: Processing S3 data with Apache Flink

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Processing-S3-data-with-Apache-Flink-tp3046p3177.html

@Konstantin (2) : Can you try the workaround described by Robert, with the "s3n" file system scheme?

We are removing the custom S3 connector now, simply reusing Hadoop's S3 connector for all cases.

@Kostia:

You are right, there should be no broken stuff that is not clearly marked as "beta". For the S3 connector, that was a problem in the testing on our side and should not have happened.

In general, you can assume that stuff in "flink-contrib" is in beta status, as well as the stuff in "flink-staging" (although much of the staging stuff will graduate with the next release). All code not in these projects should be well functioning. We test a lot, so there should be not many broken cases like that S3 connector.

 

Greetings,

Stephan

On Wed, Oct 14, 2015 at 11:44 AM, Ufuk Celebi <[hidden email]> wrote:

> On 10 Oct 2015, at 22:59, snntr <[hidden email]> wrote:
>
> Hey everyone,
>
> I was having the same problem with S3 and found this thread very useful.
> Everything works fine now, when I start Flink from my IDE, but when I run
> the jar in local mode I keep getting
>
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
> must be specified as the username or password (respectively) of a s3n URL,
> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
> properties (respectively).
>
> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
> machine with the required properties. What am I missing?
>
> Any advice is highly appreciated ;)

This looks like a problem with picking up the Hadoop config. Can you look into the logs to check whether the configuration is picked up? Change the log settings to DEBUG in log/log4j.properties for this. And can you provide the complete stack trace?

– Ufuk