Hi all,
I’m currently trying to use Scala to setup a simple Kafka consumer that receive JSON formatted events and then just send them to Elasticsearch. This is the first step and after I want to add some processing logic.
My code works well but interesting fields form my JSON formatted events are under the key « value ». So I tried to use a map function to get fields under « value » and send them to ES.
But when I try to use this map function I always get a ClassCastException error.
Additional Information :
• My Job run on a Flink Kubernetes Application cluster with a standalone job.
• If I remove the map function, everything is working fine and I can find my events in Elasticsearch
• I tried to replace to ‘JSONKeyValueDeserializationSchema’ by ’SimpleStringSchema’ and then use a mapper in my map function but I still have the same issue.
• I’m using Scala version 2.12.12
Here is my code :
// Create the Flink Kafka connector
val flinkKafkaConsumer = new FlinkKafkaConsumer(topics, new JSONKeyValueDeserializationSchema(false), kafkaProperties)
flinkKafkaConsumer.setStartFromEarliest()
val eventsStream: DataStream[ObjectNode] = env.addSource(flinkKafkaConsumer)
.name("Event Stream : Kafka consumer")
.map(event => event.get("value").deepCopy())
eventsStream.map(_.toString).addSink(esSinkBuilder.build()).name("Event Stream : Elasticsearch Stream")
// execute program
env.execute(« Kafka to ES")
Here is the error :
[nybble-jobmanager] org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
[nybble-jobmanager] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:665) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:447) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
[nybble-jobmanager] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
[nybble-jobmanager] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
[nybble-jobmanager] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
[nybble-jobmanager] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:306) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) ~[flink-runtime_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [scala-library-2.12.12.jar:?]
[nybble-jobmanager] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [scala-library-2.12.12.jar:?]
[nybble-jobmanager] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [scala-library-2.12.12.jar:?]
[nybble-jobmanager] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [scala-library-2.12.12.jar:?]
[nybble-jobmanager] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [scala-library-2.12.12.jar:?]
[nybble-jobmanager] at akka.actor.Actor.aroundReceive(Actor.scala:517) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [akka-actor_2.12-2.5.21.jar:2.5.21]
[nybble-jobmanager] Caused by: java.lang.ClassCastException: class org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode cannot be cast to class scala.runtime.Nothing$ (org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode and scala.runtime.Nothing$ are in unnamed module of loader 'app')
[nybble-jobmanager] at io.nybble.NybbleMain$.$anonfun$main$4(NybbleMain.scala:138) ~[?:?]
[nybble-jobmanager] at org.apache.flink.streaming.api.scala.DataStream$$anon$4.map(DataStream.scala:625) ~[flink-streaming-scala_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:322) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:426) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:365) ~[flink-connector-kafka_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:183) ~[flink-connector-kafka_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142) ~[flink-connector-kafka_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826) ~[flink-connector-kafka_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
[nybble-jobmanager] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:241) ~[flink-streaming-java_2.12-1.12.1.jar:1.12.1]
Does anyone has already had this kind of issue ? I think I’ve missed something even if it’s a really basic use-case and code, I’m beginner in Scala.
Thanks in advance for your help !
Sebastien
Locked
