Re: Integration Alluxio and Flink

Posted by Andrea Sella on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Integration-Alluxio-and-Flink-tp5498p5533.html

Hi Till,

I've tried your suggestion (source-code) and now it throws: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found. 
The core-site.xml has been set correctly and into the alluxio-wordcount jar is present alluxio-client. Do I need to specify the hadoop configuration via code or core-site.xml is enough?

Thank you again,
Andrea

2016-03-14 17:28 GMT+01:00 Till Rohrmann <[hidden email]>:

Hi Andrea,

the problem won’t be netty-all but netty, I suspect. Flink is using version 3.8 whereas alluxio-core-client uses version 3.2.2. I think you have to exclude or shade this dependency away.

Cheers,
Till

​

On Mon, Mar 14, 2016 at 5:12 PM, Andrea Sella <[hidden email]> wrote:

Hi Till,
I tried to downgrade the Alluxio's netty version from 4.0.28.Final to 4.0.27.Final to align Flink and Alluxio dependencies. First of all, Flink 1.0.0 uses 4.0.27.Final, is it correct? Btw it doesn't work, same error as above.

BR,
Andrea

2016-03-14 15:30 GMT+01:00 Till Rohrmann <[hidden email]>:

Yes it seems as if you have a netty version conflict. Maybe the alluxio-core-client.jar pulls in an incompatible netty version. Could you check whether this is the case? But maybe you also have another dependencies which pulls in a wrong netty version, since the Alluxio documentation indicates that it works with Flink (but I cannot tell for which version).

Cheers,

Till

On Mon, Mar 14, 2016 at 3:18 PM, Andrea Sella <[hidden email]> wrote:

Hi to all,

I'm trying to integrate Alluxio and Apache Flink, I followed Running Flink on Alluxio to setup Flink.

I tested in local mode executing: 

bin/flink run ./examples/batch/WordCount.jar --input alluxio:///flink/README.txt

But I've faced a TimeoutException, I attach the logs. It seems the trouble is due to netty dependencies-conflict.

Thank you,
Andrea