Kinesis connector - Jackson issue

[Tomasz Dobrzycki]

Hi guys,

I'm working with Kinesis connector and currently trying to solve a
bizarre issue.
I had problems with Kinesis and httpcomponents which I was able to
solve using steps shown in:
https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc

That did the trick and I am able to run my code successfully via
IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
messages just fine.

Unfortunately that is not true for running Flink via command line
scripts. I get this error when running start-local.sh into flink run
... :
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.dataformat.cbor.CBORFactory

I have built my Kinesis connector and installed it via mvn install. Am
I missing some steps? I'm assuming that my code is fine given that I'm
able to run it through IntelliJ.

Anyone faced this problem or maybe some solution comes to your mind?

Cheers
Tomasz

[Tzu-Li (Gordon) Tai]

Hi Tomasz,

Yes, dependency clashes may surface when executing actual job runs on clusters.

A few things to probably check first:

- Have you built Flink or the Kinesis connector with Maven version 3.3 or above? If yes, try using a lower version, as 3.3+ results in some shading issues when used to build Flink.

- I’m not sure if the Kinesis client has a Jackson dependency, but you could also try checking if your application pulls in a conflicting Jackson version (with Flink, which uses 2.7.4) via some other dependency.

Cheers,

Gordon

On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki ([hidden email]) wrote:

Hi guys,

I'm working with Kinesis connector and currently trying to solve a
bizarre issue.
I had problems with Kinesis and httpcomponents which I was able to
solve using steps shown in:
https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc

That did the trick and I am able to run my code successfully via
IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
messages just fine.

Unfortunately that is not true for running Flink via command line
scripts. I get this error when running start-local.sh into flink run
... :
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.dataformat.cbor.CBORFactory

I have built my Kinesis connector and installed it via mvn install. Am
I missing some steps? I'm assuming that my code is fine given that I'm
able to run it through IntelliJ.

Anyone faced this problem or maybe some solution comes to your mind?

Cheers
Tomasz

[Tomasz Dobrzycki]

Hi Gordon,

Thanks for your answer.
- I've built it with Maven 3.2.5
- I am using Jackson in my application (version 2.7.4)

Something that I have noticed when building Kinesis connector is that
it excludes jackson:
[INFO] Excluding
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
from the shaded jar.
even though I can't find any mention of that in it's pom.xml.

Cheers,
Tomasz

On 26 September 2017 at 15:43, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

> Hi Tomasz,
>
> Yes, dependency clashes may surface when executing actual job runs on
> clusters.
>
> A few things to probably check first:
> - Have you built Flink or the Kinesis connector with Maven version 3.3 or
> above? If yes, try using a lower version, as 3.3+ results in some shading
> issues when used to build Flink.
> - I’m not sure if the Kinesis client has a Jackson dependency, but you could
> also try checking if your application pulls in a conflicting Jackson version
> (with Flink, which uses 2.7.4) via some other dependency.
>
> Cheers,
> Gordon
>
>
> On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki
> ([hidden email]) wrote:
>
> Hi guys,
>
> I'm working with Kinesis connector and currently trying to solve a
> bizarre issue.
> I had problems with Kinesis and httpcomponents which I was able to
> solve using steps shown in:
> https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
>
> That did the trick and I am able to run my code successfully via
> IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
> messages just fine.
>
> Unfortunately that is not true for running Flink via command line
> scripts. I get this error when running start-local.sh into flink run
> ... :
> Caused by: java.lang.ClassNotFoundException:
> com.fasterxml.jackson.dataformat.cbor.CBORFactory
>
> I have built my Kinesis connector and installed it via mvn install. Am
> I missing some steps? I'm assuming that my code is fine given that I'm
> able to run it through IntelliJ.
>
> Anyone faced this problem or maybe some solution comes to your mind?
>
> Cheers
> Tomasz

[Tzu-Li (Gordon) Tai]

Ah, I see.

Are you using the Flink quickstart template to build your application?

I think exclusion is defined in the pom.xml of that archetype.

Just above the exclusion I do see this message:

“WARNING: You have to remove these excludes if your code relies on other version of these dependencies."

On 26 September 2017 at 5:27:47 PM, Tomasz Dobrzycki ([hidden email]) wrote:

Hi Gordon,

Thanks for your answer.
- I've built it with Maven 3.2.5
- I am using Jackson in my application (version 2.7.4)

Something that I have noticed when building Kinesis connector is that
it excludes jackson:
[INFO] Excluding
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
from the shaded jar.
even though I can't find any mention of that in it's pom.xml.

Cheers,
Tomasz

On 26 September 2017 at 15:43, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

> Hi Tomasz,
>
> Yes, dependency clashes may surface when executing actual job runs on
> clusters.
>
> A few things to probably check first:
> - Have you built Flink or the Kinesis connector with Maven version 3.3 or
> above? If yes, try using a lower version, as 3.3+ results in some shading
> issues when used to build Flink.
> - I’m not sure if the Kinesis client has a Jackson dependency, but you could
> also try checking if your application pulls in a conflicting Jackson version
> (with Flink, which uses 2.7.4) via some other dependency.
>
> Cheers,
> Gordon
>
>
> On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki
> ([hidden email]) wrote:
>
> Hi guys,
>
> I'm working with Kinesis connector and currently trying to solve a
> bizarre issue.
> I had problems with Kinesis and httpcomponents which I was able to
> solve using steps shown in:
> https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
>
> That did the trick and I am able to run my code successfully via
> IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
> messages just fine.
>
> Unfortunately that is not true for running Flink via command line
> scripts. I get this error when running start-local.sh into flink run
> ... :
> Caused by: java.lang.ClassNotFoundException:
> com.fasterxml.jackson.dataformat.cbor.CBORFactory
>
> I have built my Kinesis connector and installed it via mvn install. Am
> I missing some steps? I'm assuming that my code is fine given that I'm
> able to run it through IntelliJ.
>
> Anyone faced this problem or maybe some solution comes to your mind?
>
> Cheers
> Tomasz

[Tomasz Dobrzycki]

Yes I am using quickstart template. I have removed the exclusions for
jackson: core, databind and annotations.

On 26 September 2017 at 16:36, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

> Ah, I see.
>
> Are you using the Flink quickstart template to build your application?
> I think exclusion is defined in the pom.xml of that archetype.
>
> Just above the exclusion I do see this message:
> “WARNING: You have to remove these excludes if your code relies on other
> version of these dependencies."
>
>
> On 26 September 2017 at 5:27:47 PM, Tomasz Dobrzycki
> ([hidden email]) wrote:
>
> Hi Gordon,
>
> Thanks for your answer.
> - I've built it with Maven 3.2.5
> - I am using Jackson in my application (version 2.7.4)
>
> Something that I have noticed when building Kinesis connector is that
> it excludes jackson:
> [INFO] Excluding
> com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
> from the shaded jar.
> even though I can't find any mention of that in it's pom.xml.
>
> Cheers,
> Tomasz
>
> On 26 September 2017 at 15:43, Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>> Hi Tomasz,
>>
>> Yes, dependency clashes may surface when executing actual job runs on
>> clusters.
>>
>> A few things to probably check first:
>> - Have you built Flink or the Kinesis connector with Maven version 3.3 or
>> above? If yes, try using a lower version, as 3.3+ results in some shading
>> issues when used to build Flink.
>> - I’m not sure if the Kinesis client has a Jackson dependency, but you
>> could
>> also try checking if your application pulls in a conflicting Jackson
>> version
>> (with Flink, which uses 2.7.4) via some other dependency.
>>
>> Cheers,
>> Gordon
>>
>>
>> On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki
>> ([hidden email]) wrote:
>>
>> Hi guys,
>>
>> I'm working with Kinesis connector and currently trying to solve a
>> bizarre issue.
>> I had problems with Kinesis and httpcomponents which I was able to
>> solve using steps shown in:
>>
>> https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
>>
>> That did the trick and I am able to run my code successfully via
>> IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
>> messages just fine.
>>
>> Unfortunately that is not true for running Flink via command line
>> scripts. I get this error when running start-local.sh into flink run
>> ... :
>> Caused by: java.lang.ClassNotFoundException:
>> com.fasterxml.jackson.dataformat.cbor.CBORFactory
>>
>> I have built my Kinesis connector and installed it via mvn install. Am
>> I missing some steps? I'm assuming that my code is fine given that I'm
>> able to run it through IntelliJ.
>>
>> Anyone faced this problem or maybe some solution comes to your mind?
>>
>> Cheers
>> Tomasz

[Tomasz Dobrzycki]

Hi guys,

I was able to solve the issue. I deleted all my Flink distributions
and followed these steps:

1) Clone Flink source (because I'm building Flink with Kinesis connector)
2) Checkout to release-1.3.1 (that's the version of Flink on EMR)
3) mvn clean install -Pinclude-kinesis -DskipTests (using Maven 3.2.5
for that so no need to do it in flink-dist again)
4) go to application folder
5) mvn clean package
- 'flink run' with file input works fine
- IntelliJ with file input works fine
- !! Caused by: java.lang.IllegalStateException: Socket not created by
this factory when running IntelliJ or 'flink run' with Kinesis input

6) follow the changes in
https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
(need to apply them to pom.xml file of
flink-connectors/flink-connector-kinesis) !! important note !! follow
all of the including deletion of aws shading
- now the code should run via IntelliJ and 'flink run'

The trick was to apply all the changes from that PR.
Thanks for all the help and suggestions.
Oh, and Bowen Li - you're the champ mate, thanks ;)

Cheers,
Tomasz

On 26 September 2017 at 17:05, Tomasz Dobrzycki
<[hidden email]> wrote:

> Yes I am using quickstart template. I have removed the exclusions for
> jackson: core, databind and annotations.
>
> On 26 September 2017 at 16:36, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
>> Ah, I see.
>>
>> Are you using the Flink quickstart template to build your application?
>> I think exclusion is defined in the pom.xml of that archetype.
>>
>> Just above the exclusion I do see this message:
>> “WARNING: You have to remove these excludes if your code relies on other
>> version of these dependencies."
>>
>>
>> On 26 September 2017 at 5:27:47 PM, Tomasz Dobrzycki
>> ([hidden email]) wrote:
>>
>> Hi Gordon,
>>
>> Thanks for your answer.
>> - I've built it with Maven 3.2.5
>> - I am using Jackson in my application (version 2.7.4)
>>
>> Something that I have noticed when building Kinesis connector is that
>> it excludes jackson:
>> [INFO] Excluding
>> com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
>> from the shaded jar.
>> even though I can't find any mention of that in it's pom.xml.
>>
>> Cheers,
>> Tomasz
>>
>> On 26 September 2017 at 15:43, Tzu-Li (Gordon) Tai <[hidden email]>
>> wrote:
>>> Hi Tomasz,
>>>
>>> Yes, dependency clashes may surface when executing actual job runs on
>>> clusters.
>>>
>>> A few things to probably check first:
>>> - Have you built Flink or the Kinesis connector with Maven version 3.3 or
>>> above? If yes, try using a lower version, as 3.3+ results in some shading
>>> issues when used to build Flink.
>>> - I’m not sure if the Kinesis client has a Jackson dependency, but you
>>> could
>>> also try checking if your application pulls in a conflicting Jackson
>>> version
>>> (with Flink, which uses 2.7.4) via some other dependency.
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>> On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki
>>> ([hidden email]) wrote:
>>>
>>> Hi guys,
>>>
>>> I'm working with Kinesis connector and currently trying to solve a
>>> bizarre issue.
>>> I had problems with Kinesis and httpcomponents which I was able to
>>> solve using steps shown in:
>>>
>>> https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
>>>
>>> That did the trick and I am able to run my code successfully via
>>> IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
>>> messages just fine.
>>>
>>> Unfortunately that is not true for running Flink via command line
>>> scripts. I get this error when running start-local.sh into flink run
>>> ... :
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.fasterxml.jackson.dataformat.cbor.CBORFactory
>>>
>>> I have built my Kinesis connector and installed it via mvn install. Am
>>> I missing some steps? I'm assuming that my code is fine given that I'm
>>> able to run it through IntelliJ.
>>>
>>> Anyone faced this problem or maybe some solution comes to your mind?
>>>
>>> Cheers
>>> Tomasz