(DEPRECATED) Apache Flink User Mailing List archive.

JAXB Classloading errors when using PMML Library (AWS EMR Flink 1.4.2)

Classic

List

Threaded

2 messages Options

Sameer Wadkar

JAXB Classloading errors when using PMML Library (AWS EMR Flink 1.4.2)

Hi,

I am using PMML dependency as below to execute ML models at prediction time within a Flink Map operator

<groupId>org.jpmml</groupId>

<artifactId>pmml-evaluator</artifactId>

<groupId>javax.xml.bind</groupId>

</exclusion>

<groupId>org.glassfish.jaxb</groupId>

<artifactId>jaxb-runtime</artifactId>

</exclusion>

<artifactId>guava</artifactId>

<groupId>com.google.guava</groupId>

</exclusion>

</exclusions>

</dependency>

Environment is EMR, OpenJDK 1.8 and Flink 1.4.2. My programs run fine in my Eclipse Development environment. However when we deploy on the cluster we get Classloading exceptions which are primarily due to the PMML classes loaded via the Flink Classloader while the JAXB classes are loaded by the boot classloader. Also the problem seems like the version of the jaxb classes referenced within the PMML library is different from the ones loaded by the open JDK.

For example I keep getting this type of error. I have also listed another error after this which is linked to not being able to use reflection and unsafe library to set private instances within the PMML class instance using JAXB Unmarshaller. -

java.lang.LinkageError: loader constraint violation: when resolving interface method "javax.xml.bind.Unmarshaller.unmarshal(Ljavax/xml/transform/Source;)Ljava/lang/Object;" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of the current class, com/comcast/mlarche/featurecreationflows/xreerrors/MyJAXBUtil, and the class loader (instance of <bootloader>) for the method's defining class, javax/xml/bind/Unmarshaller, have different Class objects for the type javax/xml/transform/Source used in the signature

at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil.unmarshal(MyJAXBUtil.java:52)

at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil.unmarshalPMML(MyJAXBUtil.java:38)

at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution.getMiningModelEvaluator(PMMLModelExecution.java:67)

at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution.predict(PMMLModelExecution.java:126)

at com.comcast.mlarche.featurecreationflows.xreerrors.XreErrorModelsPredictionServiceService.predict(XreErrorModelsPredictionServiceService.java:61)

at com.comcast.mlarche.featurecreationflows.xreerrors.XreErrorModelsPredictionServiceService.predictSystemRefresh(XreErrorModelsPredictionServiceService.java:44)

at com.comcast.mlarche.featurecreationflows.xreerrors.XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:46)

at com.comcast.mlarche.featurecreationflows.xreerrors.XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:17)

at org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)

at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:549)

at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:524)

at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:504)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:830)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:808)

at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)

at com.comcast.mlarche.featurecreationflows.xreerrors.XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:65)

at com.comcast.mlarche.featurecreationflows.xreerrors.XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:20)

at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableWindowFunction.process(InternalIterableWindowFunction.java:44)

at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableWindowFunction.process(InternalIterableWindowFunction.java:32)

at org.apache.flink.streaming.runtime.operators.windowing.EvictingWindowOperator.emitWindowContents(EvictingWindowOperator.java:357)

at org.apache.flink.streaming.runtime.operators.windowing.EvictingWindowOperator.processElement(EvictingWindowOperator.java:218)

at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:207)

at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:264)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)

at java.lang.Thread.run(Thread.java:748)

Finally when I started using InputStream based constructor (so as not to use SAXSource classe) below is the last error I got when I finally got as deep within the library as possible without using any util classes.

java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set org.xml.sax.Locator field org.dmg.pmml.PMMLObject.locator to org.xml.sax.helpers.LocatorImpl

at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution.getMiningModelEvaluator(PMMLModelExecution.java:81)

at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution.predict(PMMLModelExecution.java:129)