elastic-search connector for 1.3.1 not working for local environemnt

[Debski]

Today I added flink-connector-ealsticsearch to my project and when I started flink engine using createLocalEnvironmentWithWebUI method I got following exception

xception in thread "main" java.lang.IllegalArgumentException
        at org.apache.flink.shaded.org.objectweb.asm.ClassVisitor.<init>(Unknown Source)
        at org.apache.flink.shaded.org.objectweb.asm.ClassVisitor.<init>(Unknown Source)
        at org.apache.flink.api.scala.InnerClosureFinder.<init>(ClosureCleaner.scala:279)
        at org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:95)
        at org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:115)
        at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.scalaClean(StreamExecutionEnvironment.scala:670)
        at org.apache.flink.streaming.api.scala.DataStream.clean(DataStream.scala:1061)
        at org.apache.flink.streaming.api.scala.DataStream.map(DataStream.scala:564)
        at Test$.main(Test.scala:61)
        at Test.main(Test.scala)

I found out that someone had similar problem in the past: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/asm-IllegalArgumentException-with-1-0-0-td5411.html and problem seems to be with shaded asm dependency in Flink jars.

I've found out that there will be some improvements soon to this situation: https://issues.apache.org/jira/browse/FLINK-6529 but currently it seems that ES connector is not very usable during development.

What surprises me is that it seems that asm dependency shaded in ES connector is from one of ASM 4.X.X releases because of contents of ClassVisitor constructor:

public ClassVisitor(int var1, ClassVisitor var2) {
        if(var1 != 262144) {
            throw new IllegalArgumentException();
        } else {
            this.api = var1;
            this.cv = var2;
        }
    }

and in 5.X.X versions it looks like

    public ClassVisitor(int var1, ClassVisitor var2) {
        if(var1 != 262144 && var1 != 327680) {
            throw new IllegalArgumentException();
        } else {
            this.api = var1;
            this.cv = var2;
        }
    }

Those magic numbers are in fact constants from ASM called ASM4 (262144) and ASM5 (327680).

I grepped through GitHub repo and there are no traces of ASM 4.X.X anywhere so I suspect that there is something wrong with published ES connector jar.

[Debski]

I've found out the source of the problem when I build flink locally. elastic-search base depends on (by default) ES version 1.7.1 that depends on asm 4.1 and that version is shaded to elasticsearch-base-jar. I tried to set elasticsearch.version property in Maven to 5.1.2 (the same as elasticsearch5 connector) but then elasticsearch-base does not compile:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project flink-connector-elasticsearch-base_2.11: Compilation failure
[ERROR] /home/adebski/Downloads/flink-release-1.3.1/flink-connectors/flink-connector-elasticsearch-base/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBaseTest.java:[491,92] no suitable constructor found for BulkItemResponse(int,java.lang.String,org.elasticsearch.action.ActionResponse)
[ERROR] constructor org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.DocWriteResponse) is not applicable
[ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot be converted to org.elasticsearch.action.DocWriteResponse)
[ERROR] constructor org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.bulk.BulkItemResponse.Failure) is not applicable
[ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot be converted to org.elasticsearch.action.bulk.BulkItemResponse.Failure)

[Aljoscha Krettek]

Hi,

Thanks a lot for figuring this out! I opened a Jira issue: https://issues.apache.org/jira/browse/FLINK-7133

Best,

Aljoscha 

On 7. Jul 2017, at 10:53, Debski <[hidden email]> wrote:

I've found out the source of the problem when I build flink locally.
elastic-search base depends on (by default) ES version 1.7.1 that depends on
asm 4.1 and that version is shaded to elasticsearch-base-jar. I tried to set
elasticsearch.version property in Maven to 5.1.2 (the same as elasticsearch5
connector) but then elasticsearch-base does not compile:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project flink-connector-elasticsearch-base_2.11:
Compilation failure
[ERROR]
/home/adebski/Downloads/flink-release-1.3.1/flink-connectors/flink-connector-elasticsearch-base/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBaseTest.java:[491,92]
no suitable constructor found for
BulkItemResponse(int,java.lang.String,org.elasticsearch.action.ActionResponse)
[ERROR] constructor
org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.DocWriteResponse)
is not applicable
[ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot
be converted to org.elasticsearch.action.DocWriteResponse)
[ERROR] constructor
org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.bulk.BulkItemResponse.Failure)
is not applicable
[ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot
be converted to org.elasticsearch.action.bulk.BulkItemResponse.Failure)






--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/elastic-search-connector-for-1-3-1-not-working-for-local-environemnt-tp14140p14141.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.