(DEPRECATED) Apache Flink User Mailing List archive.

okio and okhttp not shaded in the Flink Uber Jar on EMR

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
austin.ce
Reply | Threaded
Open this post in threaded view
♦
♦
|

okio and okhttp not shaded in the Flink Uber Jar on EMR

austin.ce
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
austin.ce
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

austin.ce
Following up to add more info, I am building my app with maven based on the sample Flink pom.xml

My shade plugin config is:
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <!-- Run shade goal on package phase -->
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <artifactSet>
                    <excludes>
                        <exclude>org.apache.flink:force-shading</exclude>
                        <exclude>com.google.code.findbugs:jsr305</exclude>
                        <exclude>org.slf4j:*</exclude>
                        <exclude>log4j:*</exclude>
                    </excludes>
                </artifactSet>
                <filters>
                    <filter>
                        <!-- Do not copy the signatures in the META-INF folder.
                        Otherwise, this might cause SecurityExceptions when using the JAR. -->
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>${main.class}</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>

Thanks again and let me know if there is more information I can provide.

Austin

On Tue, Feb 26, 2019 at 10:59 PM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Chesnay Schepler
Short-term I'd try relocating the okio/okhttp dependencies in your jar.

I'm not too keen on adding more relocations to the hadoop jar; I can't gauge the possible side-effects.

On 27.02.2019 14:54, Austin Cawley-Edwards wrote:
Following up to add more info, I am building my app with maven based on the sample Flink pom.xml

My shade plugin config is:
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <!-- Run shade goal on package phase -->
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <artifactSet>
                    <excludes>
                        <exclude>org.apache.flink:force-shading</exclude>
                        <exclude>com.google.code.findbugs:jsr305</exclude>
                        <exclude>org.slf4j:*</exclude>
                        <exclude>log4j:*</exclude>
                    </excludes>
                </artifactSet>
                <filters>
                    <filter>
                        <!-- Do not copy the signatures in the META-INF folder.
                        Otherwise, this might cause SecurityExceptions when using the JAR. -->
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>${main.class}</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>

Thanks again and let me know if there is more information I can provide.

Austin

On Tue, Feb 26, 2019 at 10:59 PM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
Gary Yao-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Gary Yao-4
In reply to this post by austin.ce
Hi,

Actually Flink's inverted class loading feature was designed to mitigate
problems with different versions of libraries that are not compatible with
each other [1]. You may want to debug why it does not work for you.

You can also try to use the Hadoop free Flink distribution, and export the
HADOOP_CLASSPATH variable [2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths

On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
austin.ce
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

austin.ce
Thanks Gary,

I will try to look into why the child-first strategy seems to have failed for this dependency.

Best,
Austin

On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <[hidden email]> wrote:
Hi,

Actually Flink's inverted class loading feature was designed to mitigate
problems with different versions of libraries that are not compatible with
each other [1]. You may want to debug why it does not work for you.

You can also try to use the Hadoop free Flink distribution, and export the
HADOOP_CLASSPATH variable [2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths

On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
Gary Yao-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Gary Yao-4
Hi Austin,

Are you running your job detached in a per-job cluster? In that case inverted
class loading does not work. This is because we add the user jar to the system
class path, and there is no dynamic class loading involved at the moment [1].

You can try the YARN session mode, or – as Chesnay already suggested – shade
the dependency on your side.

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink


On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards <[hidden email]> wrote:
Thanks Gary,

I will try to look into why the child-first strategy seems to have failed for this dependency.

Best,
Austin

On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <[hidden email]> wrote:
Hi,

Actually Flink's inverted class loading feature was designed to mitigate
problems with different versions of libraries that are not compatible with
each other [1]. You may want to debug why it does not work for you.

You can also try to use the Hadoop free Flink distribution, and export the
HADOOP_CLASSPATH variable [2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths

On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
austin.ce
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

austin.ce
Hi Gary,

No, I am running a YARN session (which I start with: flink-yarn-session --slots 4 --taskManagerMemory 16GB --jobManagerMemory 3GB --detached) and submit jobs through the REST interface. Thank you for the tips - I will probably shade it on my side. Is there an official location that the uber jar dependencies are documented that I can reference for future dependency additions?

Best,
Austin

On Thu, Feb 28, 2019 at 7:13 AM Gary Yao <[hidden email]> wrote:
Hi Austin,

Are you running your job detached in a per-job cluster? In that case inverted
class loading does not work. This is because we add the user jar to the system
class path, and there is no dynamic class loading involved at the moment [1].

You can try the YARN session mode, or – as Chesnay already suggested – shade
the dependency on your side.

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink


On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards <[hidden email]> wrote:
Thanks Gary,

I will try to look into why the child-first strategy seems to have failed for this dependency.

Best,
Austin

On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <[hidden email]> wrote:
Hi,

Actually Flink's inverted class loading feature was designed to mitigate
problems with different versions of libraries that are not compatible with
each other [1]. You may want to debug why it does not work for you.

You can also try to use the Hadoop free Flink distribution, and export the
HADOOP_CLASSPATH variable [2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths

On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Chesnay Schepler
There is no (accurate) reference of included dependencies for the flink-shaded-hadoop uber jars. The contained NOTICE file is a good starting point, but for the time being we're using a generalized version that we apply to all hadoop versions (so some things may be missing).

I believe for hadoop 2.8.4 the following dependencies are contained, based on the output of the maven-shade-plugin:
    com.github.stephenc.jcip:jcip-annotations:1.0-1
    com.google.code.gson:gson:2.2.4
    com.google.guava:guava:11.0.2
    com.google.guava:guava:11.0.2
    com.google.protobuf:protobuf-java:2.5.0
    com.google.protobuf:protobuf-java:2.5.0
    com.jamesmurty.utils:java-xmlbuilder:0.4
    com.jcraft:jsch:0.1.54
    com.nimbusds:nimbus-jose-jwt:4.41.1
    com.squareup.okhttp:okhttp:2.4.0
    com.squareup.okio:okio:1.4.0
    com.sun.jersey:jersey-client:1.9
    com.thoughtworks.paranamer:paranamer:2.7
    commons-beanutils:commons-beanutils:1.9.3
    commons-cli:commons-cli:1.3.1
    commons-codec:commons-codec:1.10
    commons-collections:commons-collections:3.2.2
    commons-configuration:commons-configuration:1.7
    commons-daemon:commons-daemon:1.0.13
    commons-digester:commons-digester:1.8.1
    commons-io:commons-io:2.4
    commons-lang:commons-lang:2.6
    commons-logging:commons-logging:1.1.3
    commons-net:commons-net:3.1
    io.netty:netty-all:4.0.23.Final
    io.netty:netty-all:4.0.23.Final
    io.netty:netty:3.6.2.Final
    io.netty:netty:3.6.2.Final
    javax.activation:activation:1.1
    javax.servlet:servlet-api:2.5
    javax.xml.bind:jaxb-api:2.2.2
    javax.xml.stream:stax-api:1.0-2
    net.java.dev.jets3t:jets3t:0.9.0
    net.java.dev.jets3t:jets3t:0.9.0
    net.minidev:accessors-smart:1.2
    net.minidev:json-smart:2.3
    org.apache.avro:avro:1.8.2
    org.apache.commons:commons-compress:1.18
    org.apache.commons:commons-math3:3.5
    org.apache.curator:curator-client:2.7.1
    org.apache.curator:curator-client:2.7.1
    org.apache.curator:curator-framework:2.7.1
    org.apache.curator:curator-framework:2.7.1
    org.apache.curator:curator-recipes:2.7.1
    org.apache.curator:curator-recipes:2.7.1
    org.apache.directory.api:api-asn1-api:1.0.0-M20
    org.apache.directory.api:api-util:1.0.0-M20
    org.apache.directory.server:apacheds-i18n:2.0.0-M15
    org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
    org.apache.hadoop:hadoop-annotations:2.8.4
    org.apache.hadoop:hadoop-annotations:2.8.4
    org.apache.hadoop:hadoop-auth:2.8.4
    org.apache.hadoop:hadoop-auth:2.8.4
    org.apache.hadoop:hadoop-common:2.8.4
    org.apache.hadoop:hadoop-common:2.8.4
    org.apache.hadoop:hadoop-hdfs-client:2.8.4
    org.apache.hadoop:hadoop-hdfs-client:2.8.4
    org.apache.hadoop:hadoop-hdfs:2.8.4
    org.apache.hadoop:hadoop-hdfs:2.8.4
    org.apache.hadoop:hadoop-mapreduce-client-core:2.8.4
    org.apache.hadoop:hadoop-mapreduce-client-core:2.8.4
    org.apache.hadoop:hadoop-yarn-api:2.8.4
    org.apache.hadoop:hadoop-yarn-api:2.8.4
    org.apache.hadoop:hadoop-yarn-client:2.8.4
    org.apache.hadoop:hadoop-yarn-client:2.8.4
    org.apache.hadoop:hadoop-yarn-common:2.8.4
    org.apache.hadoop:hadoop-yarn-common:2.8.4
    org.apache.htrace:htrace-core4:4.0.1-incubating
    org.apache.httpcomponents:httpclient:4.5.3
    org.apache.httpcomponents:httpclient:4.5.3
    org.apache.httpcomponents:httpcore:4.4.6
    org.apache.httpcomponents:httpcore:4.4.6
    org.apache.zookeeper:zookeeper:3.4.10
    org.codehaus.jackson:jackson-core-asl:1.9.13
    org.codehaus.jackson:jackson-core-asl:1.9.13
    org.codehaus.jackson:jackson-jaxrs:1.9.13
    org.codehaus.jackson:jackson-jaxrs:1.9.13
    org.codehaus.jackson:jackson-mapper-asl:1.9.13
    org.codehaus.jackson:jackson-mapper-asl:1.9.13
    org.codehaus.jackson:jackson-xc:1.9.13
    org.codehaus.jackson:jackson-xc:1.9.13
    org.fusesource.leveldbjni:leveldbjni-all:1.8
    org.mortbay.jetty:jetty-sslengine:6.1.26
    org.tukaani:xz:1.5
    org.xerial.snappy:snappy-java:1.1.4
    xerces:xercesImpl:2.9.1
    xml-apis:xml-apis:1.3.04
    xmlenc:xmlenc:0.52

On 28.02.2019 15:54, Austin Cawley-Edwards wrote:
Hi Gary,

No, I am running a YARN session (which I start with: flink-yarn-session --slots 4 --taskManagerMemory 16GB --jobManagerMemory 3GB --detached) and submit jobs through the REST interface. Thank you for the tips - I will probably shade it on my side. Is there an official location that the uber jar dependencies are documented that I can reference for future dependency additions?

Best,
Austin

On Thu, Feb 28, 2019 at 7:13 AM Gary Yao <[hidden email]> wrote:
Hi Austin,

Are you running your job detached in a per-job cluster? In that case inverted
class loading does not work. This is because we add the user jar to the system
class path, and there is no dynamic class loading involved at the moment [1].

You can try the YARN session mode, or – as Chesnay already suggested – shade
the dependency on your side.

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink


On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards <[hidden email]> wrote:
Thanks Gary,

I will try to look into why the child-first strategy seems to have failed for this dependency.

Best,
Austin

On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <[hidden email]> wrote:
Hi,

Actually Flink's inverted class loading feature was designed to mitigate
problems with different versions of libraries that are not compatible with
each other [1]. You may want to debug why it does not work for you.

You can also try to use the Hadoop free Flink distribution, and export the
HADOOP_CLASSPATH variable [2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths

On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi,

I recently experienced versioning clashes with the okio and okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After investigating and talking to the okio team (see this issue), I found that both okio and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded jar. The okio team suggested that Flink should shade the uber jar to fix the issue, but I'm wondering if there is something I can do on my end to have all versions exist simultaneously.

From the issue, here are the okio contents of the uber jar:

jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio

META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class

Thank you,
Austin Cawley-Edwards
Free forum by Nabble Edit this page