11 Jul 2018 17:49:08.579 [main] INFO org.apache.flink.api.java.typeutils.TypeExtractor.isValidPojoField 1876 - class com.adaequare.etl2.batch.operator.EntityType does not contain a setter for field targetEntity 11 Jul 2018 17:49:08.583 [main] INFO org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo 1915 - Class class com.adaequare.etl2.batch.operator.EntityType cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance. 11 Jul 2018 17:49:09.027 [main] DEBUG org.apache.flink.api.java.ClosureCleaner.cleanThis0 120 - this$0 is accessed: false 11 Jul 2018 17:49:09.065 [main] INFO org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 999 - The job has 2 registered types and 0 default Kryo serializers 11 Jul 2018 17:49:09.068 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1012 - Registered Kryo types: [class com.adaequare.etl2.batch.operator.EntityType, interface java.util.List] 11 Jul 2018 17:49:09.068 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1013 - Registered Kryo with Serializers types: [] 11 Jul 2018 17:49:09.069 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1014 - Registered Kryo with Serializer Classes types: [] 11 Jul 2018 17:49:09.069 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1015 - Registered Kryo default Serializers: [] 11 Jul 2018 17:49:09.070 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1016 - Registered Kryo default Serializers Classes [] 11 Jul 2018 17:49:09.070 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1017 - Registered POJO types: [] 11 Jul 2018 17:49:09.072 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1020 - Static code analysis mode: DISABLE 11 Jul 2018 17:49:09.085 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 433 - Beginning compilation of program 'plan' 11 Jul 2018 17:49:09.095 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 442 - Using a default parallelism of 4 11 Jul 2018 17:49:09.095 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 443 - Using default data exchange mode PIPELINED 11 Jul 2018 17:49:09.120 [main] DEBUG org.apache.flink.core.fs.FileSystem.loadFileSystems 937 - Loading extension file systems via services 11 Jul 2018 17:49:09.124 [main] INFO org.apache.flink.core.fs.FileSystem.loadHadoopFsFactory 1005 - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 11 Jul 2018 17:49:09.136 [main] DEBUG org.apache.flink.api.common.io.FileInputFormat.open 810 - Opening input split file:/tmp/stream2file8333235519400351128.tmp [0,42] 11 Jul 2018 17:49:09.150 [main] DEBUG org.apache.flink.api.common.io.FileInputFormat.open 810 - Opening input split file:/tmp/stream2file8333235519400351128.tmp [21,21] { "nodes": [ { "id": 2, "type": "source", "pact": "Data Source", "contents": "at readFile(ExecutionEnvironment.java:515) (com.adaequare.etl2.batch.operator.EntityCsvInputFormat)", "parallelism": "4", "global_properties": [ { "name": "Partitioning", "value": "RANDOM_PARTITIONED" }, { "name": "Partitioning Order", "value": "(none)" }, { "name": "Uniqueness", "value": "not unique" } ], "local_properties": [ { "name": "Order", "value": "(none)" }, { "name": "Grouping", "value": "not grouped" }, { "name": "Uniqueness", "value": "not unique" } ], "estimates": [ { "name": "Est. Output Size", "value": "42.00" }, { "name": "Est. Cardinality", "value": "3.00" } ], "costs": [ { "name": "Network", "value": "0.0" }, { "name": "Disk I/O", "value": "42.00" }, { "name": "CPU", "value": "0.0" }, { "name": "Cumulative Network", "value": "0.0" }, { "name": "Cumulative Disk I/O", "value": "42.00" }, { "name": "Cumulative CPU", "value": "0.0" } ], "compiler_hints": [ { "name": "Output Size (bytes)", "value": "(none)" }, { "name": "Output Cardinality", "value": "(none)" }, { "name": "Avg. Output Record Size (bytes)", "value": "(none)" }, { "name": "Filter Factor", "value": "(none)" } ] }, { "id": 1, "type": "pact", "pact": "Map", "contents": "Map at lambda$configure$0(MapOperator.java:21)", "parallelism": "4", "predecessors": [ {"id": 2, "ship_strategy": "Forward", "exchange_mode": "PIPELINED"} ], "driver_strategy": "Map", "global_properties": [ { "name": "Partitioning", "value": "RANDOM_PARTITIONED" }, { "name": "Partitioning Order", "value": "(none)" }, { "name": "Uniqueness", "value": "not unique" } ], "local_properties": [ { "name": "Order", "value": "(none)" }, { "name": "Grouping", "value": "not grouped" }, { "name": "Uniqueness", "value": "not unique" } ], "estimates": [ { "name": "Est. Output Size", "value": "(unknown)" }, { "name": "Est. Cardinality", "value": "3.00" } ], "costs": [ { "name": "Network", "value": "0.0" }, { "name": "Disk I/O", "value": "0.0" }, { "name": "CPU", "value": "0.0" }, { "name": "Cumulative Network", "value": "0.0" }, { "name": "Cumulative Disk I/O", "value": "42.00" }, { "name": "Cumulative CPU", "value": "0.0" } ], "compiler_hints": [ { "name": "Output Size (bytes)", "value": "(none)" }, { "name": "Output Cardinality", "value": "(none)" }, { "name": "Avg. Output Record Size (bytes)", "value": "(none)" }, { "name": "Filter Factor", "value": "(none)" } ] }, { "id": 0, "type": "sink", "pact": "Data Sink", "contents": "com.adaequare.etl2.batch.operator.EntityCsvOutputFormat@3023df74", "parallelism": "1", "predecessors": [ {"id": 1, "ship_strategy": "Redistribute", "exchange_mode": "PIPELINED"} ], "global_properties": [ { "name": "Partitioning", "value": "RANDOM_PARTITIONED" }, { "name": "Partitioning Order", "value": "(none)" }, { "name": "Uniqueness", "value": "not unique" } ], "local_properties": [ { "name": "Order", "value": "(none)" }, { "name": "Grouping", "value": "not grouped" }, { "name": "Uniqueness", "value": "not unique" } ], "estimates": [ { "name": "Est. Output Size", "value": "(unknown)" }, { "name": "Est. Cardinality", "value": "3.00" } ], "costs": [ { "name": "Network", "value": "(unknown)" }, { "name": "Disk I/O", "value": "0.0" }, { "name": "CPU", "value": "0.0" }, { "name": "Cumulative Network", "value": "(unknown)" }, { "name": "Cumulative Disk I/O", "value": "42.00" }, { "name": "Cumulative CPU", "value": "0.0" } ], "compiler_hints": [ { "name": "Output Size (bytes)", "value": "(none)" }, { "name": "Output Cardinality", "value": "(none)" }, { "name": "Avg. Output Record Size (bytes)", "value": "(none)" }, { "name": "Filter Factor", "value": "(none)" } ] } ] } 11 Jul 2018 17:49:09.240 [main] INFO org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 999 - The job has 2 registered types and 0 default Kryo serializers 11 Jul 2018 17:49:09.241 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1012 - Registered Kryo types: [class com.adaequare.etl2.batch.operator.EntityType, interface java.util.List] 11 Jul 2018 17:49:09.242 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1013 - Registered Kryo with Serializers types: [] 11 Jul 2018 17:49:09.244 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1014 - Registered Kryo with Serializer Classes types: [] 11 Jul 2018 17:49:09.245 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1015 - Registered Kryo default Serializers: [] 11 Jul 2018 17:49:09.247 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1016 - Registered Kryo default Serializers Classes [] 11 Jul 2018 17:49:09.252 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1017 - Registered POJO types: [] 11 Jul 2018 17:49:09.253 [main] DEBUG org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan 1020 - Static code analysis mode: DISABLE 11 Jul 2018 17:49:09.327 [main] WARN org.apache.flink.configuration.Configuration.getRawValueFromOption 771 - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address' 11 Jul 2018 17:49:09.795 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.logging.InternalLoggerFactory.debug 71 - Using SLF4J as the default logging framework 11 Jul 2018 17:49:09.799 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.debug 76 - -Dio.netty.eventLoopThreads: 8 11 Jul 2018 17:49:09.833 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug 76 - java.nio.Buffer.address: available 11 Jul 2018 17:49:09.834 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug 76 - sun.misc.Unsafe.theUnsafe: available 11 Jul 2018 17:49:09.834 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug 71 - sun.misc.Unsafe.copyMemory: available 11 Jul 2018 17:49:09.835 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0.debug 76 - java.nio.Bits.unaligned: true 11 Jul 2018 17:49:09.836 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - Java version: 8 11 Jul 2018 17:49:09.836 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - -Dio.netty.noUnsafe: false 11 Jul 2018 17:49:09.836 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - sun.misc.Unsafe: available 11 Jul 2018 17:49:09.837 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - -Dio.netty.noJavassist: false 11 Jul 2018 17:49:09.894 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 71 - Javassist: available 11 Jul 2018 17:49:09.894 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) 11 Jul 2018 17:49:09.894 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - -Dio.netty.bitMode: 64 (sun.arch.data.model) 11 Jul 2018 17:49:09.895 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent.debug 76 - -Dio.netty.noPreferDirect: false 11 Jul 2018 17:49:09.907 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.debug 76 - -Dio.netty.noKeySetOptimization: false 11 Jul 2018 17:49:09.908 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.debug 76 - -Dio.netty.selectorAutoRebuildThreshold: 512 11 Jul 2018 17:49:09.913 [main] TRACE org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.trace 46 - Instrumented an optimized java.util.Set into: sun.nio.ch.EPollSelectorImpl@77d2e85 11 Jul 2018 17:49:09.918 [main] INFO org.apache.flink.runtime.rest.RestClient. 119 - Rest client endpoint started. 11 Jul 2018 17:49:09.920 [main] DEBUG org.apache.flink.client.program.ClusterClient.getOptimizedPlan 355 - Set parallelism 1, plan default parallelism 4 11 Jul 2018 17:49:09.921 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 433 - Beginning compilation of program 'Flink Java Job at Wed Jul 11 17:49:09 IST 2018' 11 Jul 2018 17:49:09.921 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 442 - Using a default parallelism of 4 11 Jul 2018 17:49:09.921 [main] DEBUG org.apache.flink.optimizer.Optimizer.compile 443 - Using default data exchange mode PIPELINED 11 Jul 2018 17:49:09.922 [main] DEBUG org.apache.flink.api.common.io.FileInputFormat.open 810 - Opening input split file:/tmp/stream2file8333235519400351128.tmp [0,42] 11 Jul 2018 17:49:09.923 [main] DEBUG org.apache.flink.api.common.io.FileInputFormat.open 810 - Opening input split file:/tmp/stream2file8333235519400351128.tmp [21,21] 11 Jul 2018 17:49:09.980 [main] INFO org.apache.flink.client.program.rest.RestClusterClient.submitJob 240 - Submitting job 3164bc57c90c80756a4ef962d1d8b5c7 (detached: false). 11 Jul 2018 17:49:09.981 [main] INFO org.apache.flink.client.program.rest.RestClusterClient.submitJob 316 - Requesting blob server port. 11 Jul 2018 17:49:09.994 [main] DEBUG org.apache.flink.runtime.rest.RestClient.sendRequest 156 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to localhost:6123/blobserver/port 11 Jul 2018 17:49:10.029 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector.debug 81 - -Dorg.apache.flink.shaded.netty4.io.netty.leakDetectionLevel: simple 11 Jul 2018 17:49:10.045 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadLocalRandom.debug 71 - -Dio.netty.initialSeedUniquifier: 0x24602cf608346d85 (took 0 ms) 11 Jul 2018 17:49:10.060 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil.debug 76 - -Dio.netty.allocator.type: unpooled 11 Jul 2018 17:49:10.060 [main] DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil.debug 76 - -Dio.netty.threadLocalDirectBufferSize: 65536 11 Jul 2018 17:49:10.099 [flink-rest-client-netty-1-1] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.JavassistTypeParameterMatcherGenerator.debug 76 - Generated: org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectMatcher 11 Jul 2018 17:49:10.119 [Flink-RestClusterClient-IO-thread-3] DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler.debug 76 - -Dio.netty.recycler.maxCapacity.default: 262144 11 Jul 2018 17:49:10.127 [flink-rest-client-netty-1-1] DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.Cleaner0.debug 76 - java.nio.ByteBuffer.cleaner(): available