Re: Processing S3 data with Apache Flink

Posted by snntr on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Processing-S3-data-with-Apache-Flink-tp3046p3625.html

Hi Robert,

thanks a lot, it's working now. Actually, it also says "directory" in
the description. So I should have known :/

On additional question though. If I use the flink binary for Hadoop
1.2.1 and run flink in standalone mode, should I use the *-hadoop1
dependencies even If I am not interacting with HDFS 1.x?

Cheers,

Konstantin

On 21.11.2015 14:52, Robert Metzger wrote:

> Hi,
>
> It seems that you've set the "fs.hdfs.hadoopconf" configuration
> parameter to a file. I think you have to set it the directory containing
> the configuration.
> Sorry, I know that's not very intuitive, but in Hadoop the settings for
> in different files (hdfs|yarn|core)-site.xml.
>
>
> On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi Ufuk,
>
>     sorry for not getting back to you for so long, and thanks for your
>     answer. The problem persists unfortunately. Running the job from the IDE
>     works (with core-site.xml on classpath), running it in local standalone
>     mode does not. AccessKeyID and SecretAccesKey are not found.
>
>     Attached the jobmanager log on DEBUG level. The core-site.xml is
>     definitely at the configured location.
>
>     I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to
>     run the jar in local mode. Do I have to use the Hadoop 2.x version for
>     this to work? I have put hadoop-common-2.3.jar into the flink lib
>     folder.
>
>     I don't know if it is relevant (but it seems to be related), when I run
>     the job from my IDE I get the warning:
>
>     2015-11-21 12:43:11 WARN  NativeCodeLoader:62 - Unable to load
>     native-hadoop library for your platform... using builtin-java classes
>     where applicable
>
>     Cheers and thank you,
>
>     Konstantin
>
>
>     On 14.10.2015 11:44, Ufuk Celebi wrote:
>     >
>     >> On 10 Oct 2015, at 22:59, snntr <[hidden email]
>     <mailto:[hidden email]>> wrote:
>     >>
>     >> Hey everyone,
>     >>
>     >> I was having the same problem with S3 and found this thread very
>     useful.
>     >> Everything works fine now, when I start Flink from my IDE, but
>     when I run
>     >> the jar in local mode I keep getting
>     >>
>     >> java.lang.IllegalArgumentException: AWS Access Key ID and Secret
>     Access Key
>     >> must be specified as the username or password (respectively) of a
>     s3n URL,
>     >> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
>     >> properties (respectively).
>     >>
>     >> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
>     >> machine with the required properties. What am I missing?
>     >>
>     >> Any advice is highly appreciated ;)
>     >
>     > This looks like a problem with picking up the Hadoop config. Can
>     you look into the logs to check whether the configuration is picked
>     up? Change the log settings to DEBUG in log/log4j.properties for
>     this. And can you provide the complete stack trace?
>     >
>     > – Ufuk
>     >
>     >
>
>     --
>     Konstantin Knauf * [hidden email]
>     <mailto:[hidden email]> * [phone]
>     <tel:%2B49-174-3413182>
>     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082