Hello List,
We have a Flink job running reading a Kafka topic, then sending all messages with a SOAP call. We have had a situation where that SOAP call failed every time, causing the job to be RESTARTING every few seconds.
After a few hours Flink itself terminates with an OutOfMemoryError. This means that all flink jobs are now in trouble.
I dumped the heap, and noticed that it was completely filled up with two things:
- kafka metrics
- HashMap nodes related to PublicSuffixMatcher, a part of Apache HttpClient.
This leads me to believe that the restarting somehow retains references to some old failed classes/classloaders?
Of course I will repair the root cause, the failing job, but I would also like to fix things so that Flink does not die when something like this happens. I can of course set things like the max number of retries but I do not like that: I rather have the thing retry indefinitely so that when stuff is repaired the job continues normally.
I tried to find information about how Flink loads jobs but I could not make much of it.
How can I ensure that Flink does not run out of memory like this?
We're using Flink 1.1.1 and Kafka 0.9.0.1.
Thanks for your time,
Frits
Locked
