Test failed in flink-end-to-end-tests/flink-e
Locked
 
|
Hi, I got an error when tried to compile & package Flink (version 1.12 & current master). It can be reproduced by run 'mvn clean test' under flink-end-to-end-tests/flink-end-to-end-tests-common-kafka. It seems that a necessary dependency for test scope was missing and some classes can not be found. After adding the dependency kafka-avro-serializer to the pom of flink-end-to-end-tests/flink-end-to-end-tests-common-kafka everything goes well. And I just wonder that is this a bug or I missed some local setting? Best regards. Smile Error logs attached: [INFO] --------< org.apache.flink:flink-end-to-end-tests-common-kafka >-------- [INFO] Building Flink : E2E Tests : Common Kafka 1.13-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- Downloading ... [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ flink-end-to-end-tests-common-kafka --- [INFO] Deleting /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/target [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (validate) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- spotless-maven-plugin:2.4.2:check (spotless-check) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven-version) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (ban-unsafe-snakeyaml) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (ban-unsafe-jackson) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (forbid-log4j-1) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- gmavenplus-plugin:1.8.1:execute (merge-categories) @ flink-end-to-end-tests-common-kafka --- [INFO] Using plugin classloader, includes GMavenPlus classpath. [INFO] Using Groovy 2.5.12 to perform execute. includes: org.apache.flink.tests.util.categories.Dummy excludes: [INFO] [INFO] --- directory-maven-plugin:0.1:highest-basedir (directories) @ flink-end-to-end-tests-common-kafka --- [INFO] Highest basedir set to: /Users/smile/Downloads/W/code/flink/apache/master/flink [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ flink-end-to-end-tests-common-kafka --- [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ flink-end-to-end-tests-common-kafka --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ flink-end-to-end-tests-common-kafka --- [INFO] Compiling 5 source files to /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/target/classes [INFO] /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/java/org/apache/flink/tests/util/kafka/KafkaContainerClient.java: /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/java/org/apache/flink/tests/util/kafka/KafkaContainerClient.java uses unchecked or unsafe operations. [INFO] /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/main/java/org/apache/flink/tests/util/kafka/KafkaContainerClient.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ flink-end-to-end-tests-common-kafka --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ flink-end-to-end-tests-common-kafka --- [INFO] Compiling 4 source files to /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/target/test-classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:[113,20] cannot access io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe class file for io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe not found [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 29.842 s [INFO] Finished at: 2021-01-19T18:04:39+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project flink-end-to-end-tests-common-kafka: Compilation failure [ERROR] /Users/smile/Downloads/W/code/flink/apache/master/flink/flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/SQLClientSchemaRegistryITCase.java:[113,20] cannot access io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe [ERROR] class file for io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe not found [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException |
|
Since our CI system is able to build Flink, I believe it's a local issue. Are you sure that the build is failing when you build Flink from the root directory (not calling maven from within a maven module?) On Tue, Jan 19, 2021 at 11:19 AM Smile@LETTers <[hidden email]> wrote:
|
|
|
Yes, I've tried from both the root directory and the sub module. Neither or them works. And the error messages are the same.
At 2021-01-21 23:22:12, "Robert Metzger" <[hidden email]> wrote:
|
|
|
Hi Smile, Have you used a clean checkout? I second Robert's statement considering that the dependency you're talking about is already part of flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml. It also has the correct scope set both in master and release-1.12. Matthias On Fri, Jan 22, 2021 at 10:04 AM Smile@LETTers <[hidden email]> wrote: Yes, I've tried from both the root directory and the sub module. Neither or them works. And the error messages are the same. |
|
|
Hi Matthias,
Sorry for my miss leading. I mean kafka-schema-serializer rather than kafka-avro-serializer. io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe is in kafka-schema-serializer and kafka-schema-serializer should be a dependency of kafka-avro-serializer according to their pom.xml files(see [1], [2]). I couldn't resolve a valid kafka-avro-serializer.jar in my mirror so I downloaded it manually from [3] and installed it using: mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-avro-serializer -Dversion=5.5.2 -Dpackaging=jar -Dfile=kafka-avro-serializer-5.5.2.jar After that, I tried to build Flink and got the above exceptions. Then I tried to add the dependency of kafka-schema-serializer to flink-end-to-end-tests/flink-end-to-end-tests-common-kafka/pom.xml (also manually installed it to my local maven repo) and everything went well. I also tried to remove it from the pom.xml after installing, and the exception came back. Maybe there was something wrong with the manually-installed kafka-avro-serializer? [1]. https://mvnrepository.com/artifact/io.confluent/kafka-schema-serializer/usages [2]. https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.5.2/kafka-avro-serializer-5.5.2.pom [3]. https://mvnrepository.com/artifact/io.confluent/kafka-avro-serializer At 2021-01-22 21:22:51, "Matthias Pohl" <[hidden email]> wrote:
|
|
|
Hi Smile, you missed installing the pom provided by mvnrepository.org [1]. Maven will install a basic pom if none is provided [2]. This basic pom file will not include any dependencies. You should be able to fix your problem by running your command above but adding the -DpomFile property with the pom file provided in [1]: mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-avro-serializer -Dversion=5.5.2 -Dpackaging=jar -Dfile=kafka-avro-serializer-5.5.2.jar -DpomFile=kafka-avro-serializer-5.5.2.pom On Mon, Jan 25, 2021 at 8:25 AM Smile@LETTers <[hidden email]> wrote: Hi Matthias, |
|
|
Thanks, Matthias!
I tried your suggestion and it does work. After installing kafka-avro-serializer with pom I got some more errors about io.confluent:kafka-schema-registry-parent:pom:5.5.2 and io.confluent:rest-utils-parent:pom:5.5.2 and so on. After manually installing all these dependencies with pom, the build succeeded. --------------------------------------- mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-avro-serializer -Dversion=5.5.2 -Dpackaging=jar -Dfile=kafka-avro-serializer-5.5.2.jar -DpomFile=kafka-avro-serializer-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-serializer -Dversion=5.5.2 -Dpackaging=jar -Dfile=kafka-schema-serializer-5.5.2.jar -DpomFile=kafka-schema-serializer-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=5.5.2 -Dpackaging=jar -Dfile=kafka-schema-registry-client-5.5.2.jar -DpomFile=kafka-schema-registry-client-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-parent -Dversion=5.5.2 -Dpackaging=pom -Dfile=kafka-schema-registry-parent-5.5.2.pom -DpomFile=kafka-schema-registry-parent-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=common -Dversion=5.5.2 -Dpackaging=pom -Dfile=common-5.5.2.pom -DpomFile=common-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=common-parent -Dversion=5.5.2 -Dpackaging=pom -Dfile=common-parent-5.5.2.pom -DpomFile=common-parent-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=common-config -Dversion=5.5.2 -Dpackaging=jar -Dfile=common-config-5.5.2.jar -DpomFile=common-config-5.5.2.pom mvn install:install-file -DgroupId=io.confluent -DartifactId=common-utils -Dversion=5.5.2 -Dpackaging=jar -Dfile=common-utils-5.5.2.jar -DpomFile=common-utils-5.5.2.pom At 2021-01-25 16:36:12, "Matthias Pohl" <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |