(DEPRECATED) Apache Flink User Mailing List archive.

Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Prez Cannady-3
Reply | Threaded
Open this post in threaded view
♦
♦
|

Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11

Prez Cannady-3
I’ve forked and am now experimenting with Robert Metzler’s kaka-example.


Work fine from the vanilla fork (on the master branch).  I performed my changes on branch enerscore-2.11, which includes:

1. Going from Flink 0.9 to Flink 0.10.2.
2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka 0.9).
3. Removed the shade plugin.
4. Using Scala 2.11
5. Instrumenting with spring-boot.

Currently running into this problem while attempting to invoke `mvn clean spring-boot:run`:

```
[INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @ kafka-example ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[WARNING]
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/runtime/state/CheckpointListener
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
        ... 6 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.state.CheckpointListener
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 29 more
```

My guess is that this occurs in the try block of the constructor `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T> deserializer, Properties props)`, and is probably due to me missing something in the classpath.  Any hints on how to proceed would be appreciated.


Prez Cannady  
Maximilian Michels
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11

Maximilian Michels
Hi Prez,

It appears Spring's Classloader is not set up correctly.
Unfortunately, I'm not familiar with the way Springboot works.

You added flink-connector-kafka-0.9_2.10 but also mentioned you're
using Scala 2.11. That is bound to cause troubles :)

Cheers,
Max

On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
<[hidden email]> wrote:

> I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
>
> https://github.com/OCExercise/kafka-example
>
> Work fine from the vanilla fork (on the master branch).  I performed my
> changes on branch enerscore-2.11, which includes:
>
> 1. Going from Flink 0.9 to Flink 0.10.2.
> 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> 0.9).
> 3. Removed the shade plugin.
> 4. Using Scala 2.11
> 5. Instrumenting with spring-boot.
>
> Currently running into this problem while attempting to invoke `mvn clean
> spring-boot:run`:
>
> ```
> [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> kafka-example ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> [WARNING]
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/state/CheckpointListener
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
>         ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.state.CheckpointListener
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 29 more
> ```
>
> My guess is that this occurs in the try block of the constructor
> `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> deserializer, Properties props)`, and is probably due to me missing
> something in the classpath.  Any hints on how to proceed would be
> appreciated.
>
>
> Prez Cannady
> p: 617 500 3378
> e: [hidden email]
> GH: https://github.com/opencorrelate
> LI: https://www.linkedin.com/in/revprez
>
>
>
>
>
>
>
>
>
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11

Stephan Ewen
Hi!

Can you use version 1.0.0 for everything? That should make all dependencies consistent.

Greetings,
Stephan


On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <[hidden email]> wrote:
Hi Prez,

It appears Spring's Classloader is not set up correctly.
Unfortunately, I'm not familiar with the way Springboot works.

You added flink-connector-kafka-0.9_2.10 but also mentioned you're
using Scala 2.11. That is bound to cause troubles :)

Cheers,
Max

On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
<[hidden email]> wrote:
> I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
>
> https://github.com/OCExercise/kafka-example
>
> Work fine from the vanilla fork (on the master branch).  I performed my
> changes on branch enerscore-2.11, which includes:
>
> 1. Going from Flink 0.9 to Flink 0.10.2.
> 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> 0.9).
> 3. Removed the shade plugin.
> 4. Using Scala 2.11
> 5. Instrumenting with spring-boot.
>
> Currently running into this problem while attempting to invoke `mvn clean
> spring-boot:run`:
>
> ```
> [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> kafka-example ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> [WARNING]
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/state/CheckpointListener
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
>         ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.state.CheckpointListener
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 29 more
> ```
>
> My guess is that this occurs in the try block of the constructor
> `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> deserializer, Properties props)`, and is probably due to me missing
> something in the classpath.  Any hints on how to proceed would be
> appreciated.
>
>
> Prez Cannady
> p: 617 500 3378
> e: [hidden email]
> GH: https://github.com/opencorrelate
> LI: https://www.linkedin.com/in/revprez
>
>
>
>
>
>
>
>
>
Prez Cannady-3
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Attempting to refactor Kafka Example using Flink 0.10.2 with Scala 2.11

Prez Cannady-3
I’ll give that a shot, but I should also report that as of yesterday I’ve been able to get it to work with this configuration:


My primary blocking issue seems to have been attempting to publish and subscribe to an older (~0.8.x.x) Kafka instance.

Prez Cannady  








On Mar 9, 2016, at 6:31 AM, Stephan Ewen <[hidden email]> wrote:

Hi!

Can you use version 1.0.0 for everything? That should make all dependencies consistent.

Greetings,
Stephan


On Wed, Mar 9, 2016 at 11:39 AM, Maximilian Michels <[hidden email]> wrote:
Hi Prez,

It appears Spring's Classloader is not set up correctly.
Unfortunately, I'm not familiar with the way Springboot works.

You added flink-connector-kafka-0.9_2.10 but also mentioned you're
using Scala 2.11. That is bound to cause troubles :)

Cheers,
Max

On Thu, Mar 3, 2016 at 8:02 PM, Prez Cannady
<[hidden email]> wrote:
> I’ve forked and am now experimenting with Robert Metzler’s kaka-example.
>
> https://github.com/OCExercise/kafka-example
>
> Work fine from the vanilla fork (on the master branch).  I performed my
> changes on branch enerscore-2.11, which includes:
>
> 1. Going from Flink 0.9 to Flink 0.10.2.
> 2. Using flink-connector-kafka-0.9_2.10 1.1-SNAPSHOT (which brings in Kafka
> 0.9).
> 3. Removed the shade plugin.
> 4. Using Scala 2.11
> 5. Instrumenting with spring-boot.
>
> Currently running into this problem while attempting to invoke `mvn clean
> spring-boot:run`:
>
> ```
> [INFO] --- spring-boot-maven-plugin:1.3.1.RELEASE:run (default-cli) @
> kafka-example ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/revprez/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> [WARNING]
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at
> org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:467)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/state/CheckpointListener
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at com.dataartisans.ReadFromKafka.main(ReadFromKafka.java:53)
>         ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.state.CheckpointListener
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 29 more
> ```
>
> My guess is that this occurs in the try block of the constructor
> `FlinkKafkaConsumer09(List<String> topics, KeyedDeserializationSchema<T>
> deserializer, Properties props)`, and is probably due to me missing
> something in the classpath.  Any hints on how to proceed would be
> appreciated.
>
>
> Prez Cannady
> p: 617 500 3378
> e: [hidden email]
> GH: https://github.com/opencorrelate
> LI: https://www.linkedin.com/in/revprez
>
>
>
>
>
>
>
>
>
Free forum by Nabble Edit this page