Re: Processing S3 data with Apache Flink

Posted by Ufuk Celebi on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Processing-S3-data-with-Apache-Flink-tp3046p3116.html


> On 10 Oct 2015, at 22:59, snntr <[hidden email]> wrote:
>
> Hey everyone,
>
> I was having the same problem with S3 and found this thread very useful.
> Everything works fine now, when I start Flink from my IDE, but when I run
> the jar in local mode I keep getting
>
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
> must be specified as the username or password (respectively) of a s3n URL,
> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
> properties (respectively).
>
> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
> machine with the required properties. What am I missing?
>
> Any advice is highly appreciated ;)

This looks like a problem with picking up the Hadoop config. Can you look into the logs to check whether the configuration is picked up? Change the log settings to DEBUG in log/log4j.properties for this. And can you provide the complete stack trace?

– Ufuk