Re: data flow example on cluster

Posted by Lydia Ickler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/data-flow-example-on-cluster-tp2963p3019.html

Thanks, Till!

I used the ALS from FlinkML and it works :)

Best regards,

Lydia

Am 02.10.2015 um 14:14 schrieb Till Rohrmann <[hidden email]>:

Hi Lydia,

I think the APIs of the versions 0.8-incubating-SNAPSHOT and 0.10-SNAPSHOT are not compatible. Thus, it’s not just simply setting the dependencies to 0.10-SNAPSHOT. You also have to fix the API changes. This might not be trivial. Therefore, I’d recommend you to simply use the ALS implementation which you can find in FlinkML.

Cheers,
Till

​

On Fri, Oct 2, 2015 at 2:11 PM, Robert Metzger <[hidden email]> wrote:

Lydia,

can you check the log of Flink installed on the cluster? During startup, it is writing the exact commit your 0.10-SNAPSHOT is based on.

I would recommend to check out exactly that commit locally and then build Flink locally. After that, you can rebuild your jobs jar again.

With that method, there is certainly no version mismatch.

(Thats one of the reasons why using SNAPSHOT versions on such environments is not recommended)

On Fri, Oct 2, 2015 at 2:05 PM, Stefano Bortoli <[hidden email]> wrote:

I had problems running a flink job with maven, probably there is some issue of classloading. For me worked to run a simple java command with the uberjar. So I build the jar using maven, and then run it this way

java -Xmx2g -cp target/youruberjar.jar yourclass arg1 arg2

hope it helps,

Stefano

2015-10-02 12:21 GMT+02:00 Lydia Ickler <[hidden email]>:

Hi,

I did not create anything by myself.

I just downloaded the files from here: https://github.com/tillrohrmann/flink-perf

And then executed mvn clean install -DskipTests

Then I opened the project within IntelliJ and there it works fine.

Then I exported it to the cluster that runs with 0.10-SNAPSHOT.

Am 02.10.2015 um 12:15 schrieb Stephan Ewen <[hidden email]>:

@Lydia  Did you create your POM files for your job with an 0.8.x quickstart?

Can you try to simply re-create your project's POM files with a new quickstart?

I think that the POMS between 0.8-incubating-SNAPSHOT and 0.10-SNAPSHOT may not be quite compatible any more...

On Fri, Oct 2, 2015 at 12:07 PM, Robert Metzger <[hidden email]> wrote:

Are you relying on a feature only available in 0.10-SNAPSHOT?

Otherwise, I would recommend to use the latest stable release (0.9.1) for your flink job and on the cluster.

On Fri, Oct 2, 2015 at 11:55 AM, Lydia Ickler <[hidden email]> wrote:

Hi,

but inside the pom of flunk-job is the flink version set to 0.8

<flink.version>0.8-incubating-SNAPSHOT</flink.version>

how can I change it to the newest?

<flink.version>0.10-SNAPSHOT</flink.version>

Is not working

Am 02.10.2015 um 11:48 schrieb Robert Metzger <[hidden email]>:

I think there is a version mismatch between the Flink version you've used to compile your job and the Flink version installed on the cluster.

Maven automagically pulls newer 0.10-SNAPSHOT versions every time you're building your job.

On Fri, Oct 2, 2015 at 11:45 AM, Lydia Ickler <[hidden email]> wrote:

Hi Till,

I want to execute your Matrix Completion program „ALSJoin“.

Locally it works perfect.

Now I want to execute it on the cluster with:

run -c com.github.projectflink.als.ALSJoin -cp /tmp/icklerly/flink-jobs-0.1-SNAPSHOT.jar 0 2 0.001 10 1 1

but I get the following error:

java.lang.NoSuchMethodError: org.apache.flink.api.scala.typeutils.CaseClassTypeInfo.<init>(Ljava/lang/Class;Lscala/collection/Seq;Lscala/collection/Seq;)V

I guess something like the flink-scala-0.10-SNAPSHOT.jar is missing.

How can I add that to the path?

Best regards,

Lydia