Flink 1.2-SNAPSHOT fails to initialize keyed state backend
Locked
 
|
Hi all,
I am working on an analytics project and am developing against flink 1.2-SNAPSHOT. The pipeline that I have built works; ie I can ingest data, perform operations, and output the expected result. I can also see checkpoints being written to RocksDB using Amazon S3 as the state backend. Whenever the application runs for more than approximately 5 - 10 minutes I get an RuntimeException: Could not initialize keyed state backed caused by a class not found exception, full stack trace posted below. This confuses me because for the program to have run and output data for the first 5 - 10 minutes as it does that class was certainly loaded successfully in the past. As well, I am submitting a single fat jar to flink that I am certain contains the class in question. Thank you in advance for any assistance Seth Wiesman java.lang.RuntimeException: Could not initialize keyed state backend. at org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:148) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91) at org.apache.flink.streaming.api.operators.StreamFlatMap.open(StreamFlatMap.java:41) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:407) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:283) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: com.mediamath.reporting.lib.streaming.UpsertTrigger at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1486) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1336) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503) at org.apache.flink.api.common.state.StateDescriptor.readObject(StateDescriptor.java:311) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKVStateMetaData(RocksDBKeyedStateBackend.java:653) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKeyGroupsInStateHandle(RocksDBKeyedStateBackend.java:626) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.doRestore(RocksDBKeyedStateBackend.java:609) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.<init>(RocksDBKeyedStateBackend.java:207) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreKeyedStateBackend(RocksDBStateBackend.java:269) at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:778) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:142) ... 6 more |
Re: Flink 1.2-SNAPSHOT fails to initialize keyed state backend
|
|
Hi Seth,
the 1.2-SNAPSHOT is very fragile at the moment because of multiple big changes for dynamic scaling. Maybe Stefan (in CC) has an idea what is happening with the keyed state backend here? Timo Am 27/09/16 um 16:14 schrieb swiesman: > Hi all, > > I am working on an analytics project and am developing against flink > 1.2-SNAPSHOT. The pipeline that I have built works; ie I can ingest data, > perform operations, and output the expected result. I can also see > checkpoints being written to RocksDB using Amazon S3 as the state backend. > Whenever the application runs for more than approximately 5 - 10 minutes I > get an RuntimeException: Could not initialize keyed state backed caused by a > class not found exception, full stack trace posted below. This confuses me > because for the program to have run and output data for the first 5 - 10 > minutes as it does that class was certainly loaded successfully in the past. > As well, I am submitting a single fat jar to flink that I am certain > contains the class in question. > > Thank you in advance for any assistance > > Seth Wiesman > > java.lang.RuntimeException: Could not initialize keyed state backend. > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:148) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91) > at > org.apache.flink.streaming.api.operators.StreamFlatMap.open(StreamFlatMap.java:41) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:407) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:283) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:590) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassNotFoundException: > com.mediamath.reporting.lib.streaming.UpsertTrigger > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628) > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) > at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1486) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1336) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) > at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:503) > at > org.apache.flink.api.common.state.StateDescriptor.readObject(StateDescriptor.java:311) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKVStateMetaData(RocksDBKeyedStateBackend.java:653) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.restoreKeyGroupsInStateHandle(RocksDBKeyedStateBackend.java:626) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBRestoreOperation.doRestore(RocksDBKeyedStateBackend.java:609) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.<init>(RocksDBKeyedStateBackend.java:207) > at > org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreKeyedStateBackend(RocksDBStateBackend.java:269) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:778) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.open(AbstractStreamOperator.java:142) > ... 6 more > > > > > -- > View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-2-SNAPSHOT-fails-to-initialize-keyed-state-backend-tp9208.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. -- Freundliche Grüße / Kind Regards Timo Walther Follow me: @twalthr https://www.linkedin.com/in/twalthr |
Re: Flink 1.2-SNAPSHOT fails to initialize keyed state backend
|
|
Hi! This was a temporary regression on the snapshot that has been fixed a few days ago. It should be in the snapshot repositories by now. Can you check if the problem persists if you force an update of your snapshots dependencies? Greetings, Stephan On Tue, Sep 27, 2016 at 5:04 PM, Timo Walther <[hidden email]> wrote: Hi Seth, |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |