submit job timeout
Locked
 
|
Hello, We have flink pods running in a Kubernetes environment. We succeed to submit jobs and the jobs get scheduled. Once they get scheduled, they get stuck in created status. When viewing the web UI, we see that there is enough available slots in the cluster. When looking at the jobmanager log, we see the following messages. Please see attached log. Starting Job Manager sed: cannot rename /opt/flink/conf/sedDSZDbD: Device or resource busy sed: cannot rename /opt/flink/conf/sedYfLkuC: Device or resource busy sed: cannot rename /opt/flink/conf/sednLrf4C: Device or resource busy mv: cannot move '/opt/flink/conf/flink-conf.yaml.tmp' to '/opt/flink/conf/flink-conf.yaml': Device or resource busy Starting standalonesession as a console application on host flink-jobmanager-7f4f5798fb-zwq9v. Picked up JAVA_TOOL_OPTIONS: -Dspring.application.name=FLINK-TASK-MANAGER 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration: 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JM_RESOURCE_PARAMS extraction logs: jvm_params: -Xmx469762048 -Xms469762048 -XX:MaxMetaspaceSize=268435456 logs: Picked up JAVA_TOOL_OPTIONS: -Dspring.application.name=FLINK-TASK-MANAGER WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 INFO [] - Loading configuration property: jobmanager.memory.process.size, 1024m INFO [] - Loading configuration property: taskmanager.memory.process.size, 16384m INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4 INFO [] - Loading configuration property: parallelism.default, 1 INFO [] - Loading configuration property: state.backend, rocksdb INFO [] - Loading configuration property: state.backend.rocksdb.localdir, /opt/flink/rocksdb INFO [] - Loading configuration property: state.checkpoints.dir, file:///opt/flink/checkpoints INFO [] - Loading configuration property: state.backend.incremental, true INFO [] - Loading configuration property: jobmanager.execution.failover-strategy, region INFO [] - Loading configuration property: rest.port, 8081 INFO [] - Loading configuration property: classloader.resolve-order, child-first INFO [] - Loading configuration property: metrics.reporters, prom INFO [] - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter INFO [] - Loading configuration property: metrics.reporter.prom.port, 9249 INFO [] - Loading configuration property: queryable-state.enable, true INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: query.server.port, 6125 INFO [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 1024.000mb (1073741824 bytes) INFO [] - Total Flink Memory: 576.000mb (603979776 bytes) INFO [] - JVM Heap: 448.000mb (469762048 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 192.000mb (201326592 bytes) 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneSessionClusterEntrypoint (Version: 1.11.1, Scala: 2.11, Rev:7eb514a, Date:2020-07-15T07:02:09+02:00) 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: <no hadoop dependency found> 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 11/11.0.8+10 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 433 MiBytes 2020-09-29 08:50:34,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-11 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options: 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dspring.application.name=FLINK-TASK-MANAGER 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx469762048 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms469762048 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--standalonesession-0-flink-jobmanager-7f4f5798fb-zwq9v.log 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments: 2020-09-29 08:50:34,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --configDir 2020-09-29 08:50:34,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /opt/flink/conf 2020-09-29 08:50:34,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --executionMode 2020-09-29 08:50:34,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - cluster 2020-09-29 08:50:34,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.11.1.jar:/opt/flink/lib/flink-json-1.11.1.jar:/opt/flink/lib/flink-pipeline-jobs-develop_03_0080-SNAPSHOT-all.jar:/opt/flink/lib/flink-queryable-state-runtime_2.11-1.11.1.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.11.1.jar:/opt/flink/lib/flink-table_2.11-1.11.1.jar:/opt/flink/lib/kafka-clients-2.4.1.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.11.1.jar::: 2020-09-29 08:50:34,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 2020-09-29 08:50:34,625 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT] 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1024m 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 16384m 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, rocksdb 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend.rocksdb.localdir, /opt/flink/rocksdb 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, file:///opt/flink/checkpoints 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend.incremental, true 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.port, 8081 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: classloader.resolve-order, child-first 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporters, prom 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.port, 9249 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: queryable-state.enable, true 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124 2020-09-29 08:50:34,637 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: query.server.port, 6125 2020-09-29 08:50:34,645 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneSessionClusterEntrypoint. 2020-09-29 08:50:34,670 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem. 2020-09-29 08:50:34,674 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2020-09-29 08:50:34,710 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context. 2020-09-29 08:50:34,720 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2020-09-29 08:50:34,726 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-5587551027217503506.conf. 2020-09-29 08:50:34,732 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2020-09-29 08:50:34,733 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services. 2020-09-29 08:50:34,793 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address flink-jobmanager:6123, bind address 0.0.0.0:6123. 2020-09-29 08:50:35,191 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started 2020-09-29 08:50:35,217 INFO akka.remote.Remoting [] - Starting remoting 2020-09-29 08:50:35,352 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@flink-jobmanager:6123] 2020-09-29 08:50:35,406 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@flink-jobmanager:6123 2020-09-29 08:50:35,427 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2020-09-29 08:50:35,433 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-abefc6f3-5636-442d-a922-65b6caa55290 2020-09-29 08:50:35,438 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 2020-09-29 08:50:35,469 WARN org.apache.flink.runtime.metrics.ReporterSetup [] - Multiple implementations of the same reporter were found in 'lib' and/or 'plugins' directories for org.apache.flink.metrics.prometheus.PrometheusReporterFactory. It is recommended to remove redundant reporter JARs to resolve used versions' ambiguity. 2020-09-29 08:50:35,469 WARN org.apache.flink.runtime.metrics.ReporterSetup [] - Multiple implementations of the same reporter were found in 'lib' and/or 'plugins' directories for org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory. It is recommended to remove redundant reporter JARs to resolve used versions' ambiguity. 2020-09-29 08:50:35,485 INFO org.apache.flink.metrics.prometheus.PrometheusReporter [] - Started PrometheusReporter HTTP server on port 9249. 2020-09-29 08:50:35,486 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - Reporting metrics for reporter prom of type org.apache.flink.metrics.prometheus.PrometheusReporter. 2020-09-29 08:50:35,489 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address flink-jobmanager:0, bind address 0.0.0.0:0. 2020-09-29 08:50:35,502 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started 2020-09-29 08:50:35,507 INFO akka.remote.Remoting [] - Starting remoting 2020-09-29 08:50:35,525 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@flink-jobmanager:44606] 2020-09-29 08:50:35,532 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@flink-jobmanager:44606 2020-09-29 08:50:35,546 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService . 2020-09-29 08:50:35,583 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore [] - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-3ce6c0d8-0cef-401d-b50e-8b89bfacdd7c, expiration time 3600000, maximum cache size 52428800 bytes. 2020-09-29 08:50:35,623 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 2020-09-29 08:50:35,623 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'rest.port' instead of key 'rest.bind-port' 2020-09-29 08:50:35,624 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Upload directory /tmp/flink-web-00ea6919-b182-48ac-a6d1-1f39c61d8905/flink-web-upload does not exist. 2020-09-29 08:50:35,625 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Created directory /tmp/flink-web-00ea6919-b182-48ac-a6d1-1f39c61d8905/flink-web-upload for file uploads. 2020-09-29 08:50:35,625 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Starting rest endpoint. 2020-09-29 08:50:35,908 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/flink--standalonesession-0-flink-jobmanager-7f4f5798fb-zwq9v.log 2020-09-29 08:50:35,908 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/flink--standalonesession-0-flink-jobmanager-7f4f5798fb-zwq9v.out 2020-09-29 08:50:36,031 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Rest endpoint listening at flink-jobmanager:8081 2020-09-29 08:50:36,033 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - http://flink-jobmanager:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2020-09-29 08:50:36,034 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Web frontend listening at http://flink-jobmanager:8081. 2020-09-29 08:50:36,051 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_0 . 2020-09-29 08:50:36,077 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess. 2020-09-29 08:50:36,078 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs. 2020-09-29 08:50:36,078 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs. 2020-09-29 08:50:36,083 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token 00000000000000000000000000000000 2020-09-29 08:50:36,090 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Starting the SlotManager. 2020-09-29 08:50:36,091 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 . WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtil (file:/opt/flink/lib/flink-dist_2.11-1.11.1.jar) to method java.nio.DirectByteBuffer.cleaner() WARNING: Please consider reporting this to the maintainers of org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2020-09-29 08:50:37,481 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 68fbc9f1530b27e91ad210a3b7578ba4 (akka.tcp://flink@10.39.0.1:43408/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:37,523 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 3350a9e6e6c2070dd61e38f154e10dfe (akka.tcp://flink@10.36.0.2:46789/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:37,524 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 4e3b7ae578349a8cff8a74b672e6a10a (akka.tcp://flink@10.40.0.1:36767/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:37,582 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID a15e3cd152727a1579d5a287424d0bb8 (akka.tcp://flink@10.35.0.3:34766/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:37,632 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 91aed26c766ca2a300eb4f63f56f2385 (akka.tcp://flink@10.44.0.2:36898/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:37,634 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 03aef953128bb91f24c2bd62e103796b (akka.tcp://flink@10.35.0.4:37636/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:38,152 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 9e99b95665f72980a09869a60ea9c8d5 (akka.tcp://flink@10.32.0.4:44463/user/rpc/taskmanager_0) at ResourceManager 2020-09-29 08:50:38,700 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:41,053 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission affbcfc7850aa2384821072224762245 (digestOrchestrator). 2020-09-29 08:50:41,055 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job affbcfc7850aa2384821072224762245 (digestOrchestrator). 2020-09-29 08:50:41,084 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_2 . 2020-09-29 08:50:41,090 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job digestOrchestrator (affbcfc7850aa2384821072224762245). 2020-09-29 08:50:41,141 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) for digestOrchestrator (affbcfc7850aa2384821072224762245). 2020-09-29 08:50:41,167 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job digestOrchestrator (affbcfc7850aa2384821072224762245). 2020-09-29 08:50:41,167 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2020-09-29 08:50:41,179 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 2020-09-29 08:50:41,185 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2020-09-29 08:50:41,291 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using predefined options: DEFAULT. 2020-09-29 08:50:41,291 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}. 2020-09-29 08:50:41,310 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@69d25369 for digestOrchestrator (affbcfc7850aa2384821072224762245). 2020-09-29 08:50:41,313 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job digestOrchestrator (affbcfc7850aa2384821072224762245) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@flink-jobmanager:6123/user/rpc/jobmanager_2. 2020-09-29 08:50:41,320 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job digestOrchestrator (affbcfc7850aa2384821072224762245) under job master id 00000000000000000000000000000000. 2020-09-29 08:50:41,324 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 2020-09-29 08:50:41,324 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state CREATED to RUNNING. 2020-09-29 08:50:41,334 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (6da354a4a21690b6de20cc1c4ce1ca1d) switched from CREATED to SCHEDULED. 2020-09-29 08:50:41,334 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (b95b91da1d86eb1ee163159c3a6f323c) switched from CREATED to SCHEDULED. 2020-09-29 08:50:41,334 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (eb5c78cd4fc68caa3555095f0dd8eab2) switched from CREATED to SCHEDULED. 2020-09-29 08:50:41,344 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{22f0f2983ccafab9493825d48430c6e3}] 2020-09-29 08:50:41,351 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2020-09-29 08:50:41,355 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2020-09-29 08:50:41,357 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_2 for job affbcfc7850aa2384821072224762245. 2020-09-29 08:50:41,366 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_2 for job affbcfc7850aa2384821072224762245. 2020-09-29 08:50:41,368 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2020-09-29 08:50:41,369 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{22f0f2983ccafab9493825d48430c6e3}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 08:50:41,370 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job affbcfc7850aa2384821072224762245 with allocation id 02df757ea5d6605cf354cb6a45116bcd. 2020-09-29 08:50:41,627 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:44,620 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:45,436 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:50:47,618 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:49,285 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:50:49,445 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:50:50,623 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:53,452 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:50:53,617 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:56,616 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job 974ec58fe080964a5919d9f76ff94150 not found 2020-09-29 08:50:57,458 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:50:59,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:01,464 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:51:05,471 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:51:09,284 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:09,482 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:51:19,284 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:29,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:39,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:49,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:51:59,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:09,284 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:19,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:29,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:39,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:49,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:52:59,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:09,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:19,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:29,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:39,284 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:49,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:53:59,285 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:09,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:19,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:29,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:39,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:49,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:54:59,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:55:09,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:55:19,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:55:29,284 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:55:39,283 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:55:41,356 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (b95b91da1d86eb1ee163159c3a6f323c) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 08:55:41,362 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 08:55:41,365 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 08:55:41,367 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RUNNING to RESTARTING. 2020-09-29 08:55:41,368 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (6da354a4a21690b6de20cc1c4ce1ca1d) switched from SCHEDULED to CANCELING. 2020-09-29 08:55:41,368 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (6da354a4a21690b6de20cc1c4ce1ca1d) switched from CANCELING to CANCELED. 2020-09-29 08:55:41,368 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 6da354a4a21690b6de20cc1c4ce1ca1d. 2020-09-29 08:55:41,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (eb5c78cd4fc68caa3555095f0dd8eab2) switched from SCHEDULED to CANCELING. 2020-09-29 08:55:41,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (eb5c78cd4fc68caa3555095f0dd8eab2) switched from CANCELING to CANCELED. 2020-09-29 08:55:41,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution eb5c78cd4fc68caa3555095f0dd8eab2. 2020-09-29 08:55:41,371 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution b95b91da1d86eb1ee163159c3a6f323c. 2020-09-29 08:55:41,371 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{22f0f2983ccafab9493825d48430c6e3}] timed out. 2020-09-29 08:55:51,377 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RESTARTING to RUNNING. 2020-09-29 08:55:51,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (b903398f856c17e8e8af74e61d2a20da) switched from CREATED to SCHEDULED. 2020-09-29 08:55:51,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (8d323ddd30abb56e439168237999a087) switched from CREATED to SCHEDULED. 2020-09-29 08:55:51,380 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (f72d6fa39b0d30b66341da85ca45e864) switched from CREATED to SCHEDULED. 2020-09-29 08:55:51,381 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{b2f908be96f758c40c0677b31bd4510f}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 08:55:51,381 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job affbcfc7850aa2384821072224762245 with allocation id 91b93767bd27c735500261eade54524e. 2020-09-29 08:55:55,415 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:55:59,422 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:00,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:56:03,428 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:07,436 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:10,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:56:11,444 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:15,451 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:19,458 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 08:56:20,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:56:30,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:56:40,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:56:50,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:00,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:10,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:20,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:30,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:40,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:57:50,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:00,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:10,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:20,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:30,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:40,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:58:50,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:00,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:10,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:20,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:30,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:40,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 08:59:50,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:00,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:10,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:20,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:30,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:40,085 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:50,084 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:00:51,391 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (8d323ddd30abb56e439168237999a087) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:00:51,393 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:00:51,394 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:00:51,394 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RUNNING to RESTARTING. 2020-09-29 09:00:51,395 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (b903398f856c17e8e8af74e61d2a20da) switched from SCHEDULED to CANCELING. 2020-09-29 09:00:51,395 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (b903398f856c17e8e8af74e61d2a20da) switched from CANCELING to CANCELED. 2020-09-29 09:00:51,396 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution b903398f856c17e8e8af74e61d2a20da. 2020-09-29 09:00:51,397 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (f72d6fa39b0d30b66341da85ca45e864) switched from SCHEDULED to CANCELING. 2020-09-29 09:00:51,397 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (f72d6fa39b0d30b66341da85ca45e864) switched from CANCELING to CANCELED. 2020-09-29 09:00:51,397 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution f72d6fa39b0d30b66341da85ca45e864. 2020-09-29 09:00:51,397 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 8d323ddd30abb56e439168237999a087. 2020-09-29 09:00:51,398 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{b2f908be96f758c40c0677b31bd4510f}] timed out. 2020-09-29 09:01:01,398 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RESTARTING to RUNNING. 2020-09-29 09:01:01,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (04856572e21d37b74279bfde9bd42e6d) switched from CREATED to SCHEDULED. 2020-09-29 09:01:01,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (5d72dd53831f94bc2bb36d5ddb871976) switched from CREATED to SCHEDULED. 2020-09-29 09:01:01,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (25ddc83d3e87d228e80924a0c40cc72e) switched from CREATED to SCHEDULED. 2020-09-29 09:01:01,402 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{4ee9f887b36432a2fba3428b4308dd43}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:01:01,403 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job affbcfc7850aa2384821072224762245 with allocation id 8d9dbaa56a8d8596fc1ab54fae26111e. 2020-09-29 09:01:05,432 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:09,440 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:10,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:01:13,447 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:17,455 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:20,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:01:21,464 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:25,471 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:29,478 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:01:30,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:01:40,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:01:50,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:00,706 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:10,706 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:20,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:30,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:40,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:02:50,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:00,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:10,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:20,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:30,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:40,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:03:50,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:00,706 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:10,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:20,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:30,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:40,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:04:50,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:00,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:10,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:20,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:30,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:40,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:05:50,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:00,705 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:01,409 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (5d72dd53831f94bc2bb36d5ddb871976) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:06:01,411 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:06:01,412 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:06:01,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RUNNING to RESTARTING. 2020-09-29 09:06:01,413 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (04856572e21d37b74279bfde9bd42e6d) switched from SCHEDULED to CANCELING. 2020-09-29 09:06:01,413 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (04856572e21d37b74279bfde9bd42e6d) switched from CANCELING to CANCELED. 2020-09-29 09:06:01,414 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 04856572e21d37b74279bfde9bd42e6d. 2020-09-29 09:06:01,415 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (25ddc83d3e87d228e80924a0c40cc72e) switched from SCHEDULED to CANCELING. 2020-09-29 09:06:01,415 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (25ddc83d3e87d228e80924a0c40cc72e) switched from CANCELING to CANCELED. 2020-09-29 09:06:01,415 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 25ddc83d3e87d228e80924a0c40cc72e. 2020-09-29 09:06:01,415 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 5d72dd53831f94bc2bb36d5ddb871976. 2020-09-29 09:06:01,415 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{4ee9f887b36432a2fba3428b4308dd43}] timed out. 2020-09-29 09:06:11,417 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RESTARTING to RUNNING. 2020-09-29 09:06:11,419 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (deca1139ac7635db32c4c8b8a6dc27a1) switched from CREATED to SCHEDULED. 2020-09-29 09:06:11,419 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (15333f6ff2228e8615ef92b044d3336e) switched from CREATED to SCHEDULED. 2020-09-29 09:06:11,419 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (3226cf98260d94ddc2121731e29c650f) switched from CREATED to SCHEDULED. 2020-09-29 09:06:11,420 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{35cd803d4aff57c8d7f0e25ec10d95a1}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:06:11,421 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job affbcfc7850aa2384821072224762245 with allocation id 450a195ce4908dfa8c93394ace18b968. 2020-09-29 09:06:15,452 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:16,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:19,459 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:23,465 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:26,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:27,471 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:31,477 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:35,483 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:36,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:39,495 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.39.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:06:46,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:06:56,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:06,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:16,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:26,483 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:36,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:46,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:07:56,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:06,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:16,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:26,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:36,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:46,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:08:56,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:06,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:16,483 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:26,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:36,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:46,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:09:56,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:06,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:16,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:26,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:36,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:46,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:10:56,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:11:06,482 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job affbcfc7850aa2384821072224762245 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:11:11,429 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (15333f6ff2228e8615ef92b044d3336e) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:11:11,430 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:11:11,430 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:11:11,435 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state RUNNING to FAILING. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:11:11,445 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (deca1139ac7635db32c4c8b8a6dc27a1) switched from SCHEDULED to CANCELING. 2020-09-29 09:11:11,445 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (deca1139ac7635db32c4c8b8a6dc27a1) switched from CANCELING to CANCELED. 2020-09-29 09:11:11,446 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution deca1139ac7635db32c4c8b8a6dc27a1. 2020-09-29 09:11:11,446 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (3226cf98260d94ddc2121731e29c650f) switched from SCHEDULED to CANCELING. 2020-09-29 09:11:11,446 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (3226cf98260d94ddc2121731e29c650f) switched from CANCELING to CANCELED. 2020-09-29 09:11:11,446 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 3226cf98260d94ddc2121731e29c650f. 2020-09-29 09:11:11,447 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (affbcfc7850aa2384821072224762245) switched from state FAILING to FAILED. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:11:11,448 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job affbcfc7850aa2384821072224762245. 2020-09-29 09:11:11,448 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down 2020-09-29 09:11:11,448 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 15333f6ff2228e8615ef92b044d3336e. 2020-09-29 09:11:11,448 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{35cd803d4aff57c8d7f0e25ec10d95a1}] timed out. 2020-09-29 09:11:11,451 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job affbcfc7850aa2384821072224762245 reached globally terminal state FAILED. 2020-09-29 09:11:11,480 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job digestOrchestrator(affbcfc7850aa2384821072224762245). 2020-09-29 09:11:11,484 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending SlotPool. 2020-09-29 09:11:11,484 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection cb5d30f831de374d1dd2bcdcbca1e5bd: JobManager is shutting down.. 2020-09-29 09:11:11,484 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping SlotPool. 2020-09-29 09:11:11,484 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_2 for job affbcfc7850aa2384821072224762245 from the resource manager. 2020-09-29 09:11:17,279 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 506ff1fc18b74615baf0163315a060e1 (digestOrchestrator). 2020-09-29 09:11:17,279 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 506ff1fc18b74615baf0163315a060e1 (digestOrchestrator). 2020-09-29 09:11:17,284 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 . 2020-09-29 09:11:17,284 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job digestOrchestrator (506ff1fc18b74615baf0163315a060e1). 2020-09-29 09:11:17,307 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) for digestOrchestrator (506ff1fc18b74615baf0163315a060e1). 2020-09-29 09:11:17,308 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job digestOrchestrator (506ff1fc18b74615baf0163315a060e1). 2020-09-29 09:11:17,309 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2020-09-29 09:11:17,309 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 2020-09-29 09:11:17,310 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2020-09-29 09:11:17,310 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using predefined options: DEFAULT. 2020-09-29 09:11:17,310 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}. 2020-09-29 09:11:17,317 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@5d3a1b52 for digestOrchestrator (506ff1fc18b74615baf0163315a060e1). 2020-09-29 09:11:17,317 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@flink-jobmanager:6123/user/rpc/jobmanager_3. 2020-09-29 09:11:17,318 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) under job master id 00000000000000000000000000000000. 2020-09-29 09:11:17,318 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 2020-09-29 09:11:17,318 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state CREATED to RUNNING. 2020-09-29 09:11:17,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (e06a2f4a49c99f9e9c877b67e52e02ff) switched from CREATED to SCHEDULED. 2020-09-29 09:11:17,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (273a465f8116c9909430b79e1e1f7fc2) switched from CREATED to SCHEDULED. 2020-09-29 09:11:17,319 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (6f78247d641b54bf15d47fe97b0df1ae) switched from CREATED to SCHEDULED. 2020-09-29 09:11:17,320 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{d321f76e21f6fb196e0d4dcafe1aef2c}] 2020-09-29 09:11:17,321 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2020-09-29 09:11:17,323 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2020-09-29 09:11:17,323 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_3 for job 506ff1fc18b74615baf0163315a060e1. 2020-09-29 09:11:17,325 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_3 for job 506ff1fc18b74615baf0163315a060e1. 2020-09-29 09:11:17,326 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2020-09-29 09:11:17,326 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{d321f76e21f6fb196e0d4dcafe1aef2c}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:11:17,326 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 506ff1fc18b74615baf0163315a060e1 with allocation id 46d9e0084de74a09f2bf84194308b3b9. 2020-09-29 09:11:21,369 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:25,376 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:25,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:11:29,382 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:33,387 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:35,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:11:37,393 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:41,399 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:45,406 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:11:45,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:11:55,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:05,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:15,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:25,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:35,650 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:45,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:12:55,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:05,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:15,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:25,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:35,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:45,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:13:55,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:05,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:15,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:25,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:35,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:45,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:14:55,650 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:05,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:15,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:25,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:35,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:45,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:15:55,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:16:05,650 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:16:15,649 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:16:17,326 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (273a465f8116c9909430b79e1e1f7fc2) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:16:17,328 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:16:17,328 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:16:17,328 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RUNNING to RESTARTING. 2020-09-29 09:16:17,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (e06a2f4a49c99f9e9c877b67e52e02ff) switched from SCHEDULED to CANCELING. 2020-09-29 09:16:17,329 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (e06a2f4a49c99f9e9c877b67e52e02ff) switched from CANCELING to CANCELED. 2020-09-29 09:16:17,330 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution e06a2f4a49c99f9e9c877b67e52e02ff. 2020-09-29 09:16:17,330 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (6f78247d641b54bf15d47fe97b0df1ae) switched from SCHEDULED to CANCELING. 2020-09-29 09:16:17,330 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (6f78247d641b54bf15d47fe97b0df1ae) switched from CANCELING to CANCELED. 2020-09-29 09:16:17,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 6f78247d641b54bf15d47fe97b0df1ae. 2020-09-29 09:16:17,331 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 273a465f8116c9909430b79e1e1f7fc2. 2020-09-29 09:16:17,331 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{d321f76e21f6fb196e0d4dcafe1aef2c}] timed out. 2020-09-29 09:16:27,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RESTARTING to RUNNING. 2020-09-29 09:16:27,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (120bbfa186dd20dc01e4a02587e88e13) switched from CREATED to SCHEDULED. 2020-09-29 09:16:27,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (9f67c1142fa421179b18abd19465328f) switched from CREATED to SCHEDULED. 2020-09-29 09:16:27,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (c9029bed536ed35e7bb7f21c1d2286f0) switched from CREATED to SCHEDULED. 2020-09-29 09:16:27,335 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{fa414f02d7d727a0d03cb289053d3b55}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:16:27,335 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 506ff1fc18b74615baf0163315a060e1 with allocation id e38aca4a5ff140719241d72542720442. 2020-09-29 09:16:31,361 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:35,367 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:36,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:16:39,373 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:43,379 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:46,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:16:47,385 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:51,391 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:55,399 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:16:56,017 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:06,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:16,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:26,017 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:36,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:46,017 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:17:56,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:06,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:16,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:26,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:36,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:46,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:18:56,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:06,017 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:16,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:26,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:36,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:46,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:19:56,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:06,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:16,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:26,017 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:36,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:46,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:20:56,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:06,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:16,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:26,016 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:27,341 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (9f67c1142fa421179b18abd19465328f) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:21:27,342 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:21:27,343 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:21:27,344 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RUNNING to RESTARTING. 2020-09-29 09:21:27,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (120bbfa186dd20dc01e4a02587e88e13) switched from SCHEDULED to CANCELING. 2020-09-29 09:21:27,345 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (120bbfa186dd20dc01e4a02587e88e13) switched from CANCELING to CANCELED. 2020-09-29 09:21:27,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 120bbfa186dd20dc01e4a02587e88e13. 2020-09-29 09:21:27,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (c9029bed536ed35e7bb7f21c1d2286f0) switched from SCHEDULED to CANCELING. 2020-09-29 09:21:27,346 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (c9029bed536ed35e7bb7f21c1d2286f0) switched from CANCELING to CANCELED. 2020-09-29 09:21:27,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution c9029bed536ed35e7bb7f21c1d2286f0. 2020-09-29 09:21:27,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 9f67c1142fa421179b18abd19465328f. 2020-09-29 09:21:27,348 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{fa414f02d7d727a0d03cb289053d3b55}] timed out. 2020-09-29 09:21:37,348 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RESTARTING to RUNNING. 2020-09-29 09:21:37,351 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (0a102815e69aa4aacbc95e527b12bbd4) switched from CREATED to SCHEDULED. 2020-09-29 09:21:37,351 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (1ffd12f51995a55dea70eb7830d32c1f) switched from CREATED to SCHEDULED. 2020-09-29 09:21:37,351 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (767b9120b68e34480d857ffa54afd4cd) switched from CREATED to SCHEDULED. 2020-09-29 09:21:37,352 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{fe7403fd7c847b332535364c3353f713}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:21:37,353 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 506ff1fc18b74615baf0163315a060e1 with allocation id 6b35cb173286e7c732273678866904e4. 2020-09-29 09:21:41,379 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:21:45,385 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:21:46,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:49,391 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:21:53,396 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:21:56,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:21:57,402 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:22:01,409 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:22:05,414 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:22:06,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:22:16,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:22:26,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:22:36,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:22:46,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:22:56,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:06,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:16,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:26,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:36,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:46,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:23:56,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:06,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:16,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:26,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:36,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:46,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:24:56,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:06,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:16,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:26,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:36,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:46,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:25:56,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:06,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:16,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:26,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:36,137 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:37,357 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (1ffd12f51995a55dea70eb7830d32c1f) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:26:37,359 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:26:37,359 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:26:37,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RUNNING to RESTARTING. 2020-09-29 09:26:37,360 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (0a102815e69aa4aacbc95e527b12bbd4) switched from SCHEDULED to CANCELING. 2020-09-29 09:26:37,361 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (0a102815e69aa4aacbc95e527b12bbd4) switched from CANCELING to CANCELED. 2020-09-29 09:26:37,361 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 0a102815e69aa4aacbc95e527b12bbd4. 2020-09-29 09:26:37,361 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (767b9120b68e34480d857ffa54afd4cd) switched from SCHEDULED to CANCELING. 2020-09-29 09:26:37,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (767b9120b68e34480d857ffa54afd4cd) switched from CANCELING to CANCELED. 2020-09-29 09:26:37,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 767b9120b68e34480d857ffa54afd4cd. 2020-09-29 09:26:37,362 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 1ffd12f51995a55dea70eb7830d32c1f. 2020-09-29 09:26:37,362 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{fe7403fd7c847b332535364c3353f713}] timed out. 2020-09-29 09:26:47,364 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RESTARTING to RUNNING. 2020-09-29 09:26:47,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a80519f096ad97fa4f04e1621ce835db) switched from CREATED to SCHEDULED. 2020-09-29 09:26:47,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (313e1af898f26a62bf521d6067eb2431) switched from CREATED to SCHEDULED. 2020-09-29 09:26:47,366 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (362406cb3731bda85cd6ae975c82c7a1) switched from CREATED to SCHEDULED. 2020-09-29 09:26:47,368 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{80d221a38fce34a3c528ba7d7d1e9676}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:26:47,368 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 506ff1fc18b74615baf0163315a060e1 with allocation id 8018aa0e9f18f077812dc2d9923bf078. 2020-09-29 09:26:51,402 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:26:55,409 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:26:55,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:26:59,415 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:27:03,421 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:27:05,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:27:07,427 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:27:11,433 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:27:15,439 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.36.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:27:15,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:27:25,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:27:35,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:27:45,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:27:55,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:05,752 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:15,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:25,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:35,753 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:45,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:28:55,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:05,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:15,752 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:25,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:35,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:45,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:29:55,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:05,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:15,752 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:25,752 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:35,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:45,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:30:55,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:05,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:15,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:25,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:35,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:45,751 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 506ff1fc18b74615baf0163315a060e1 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:31:47,373 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (313e1af898f26a62bf521d6067eb2431) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:31:47,375 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:31:47,375 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:31:47,376 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state RUNNING to FAILING. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:31:47,382 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a80519f096ad97fa4f04e1621ce835db) switched from SCHEDULED to CANCELING. 2020-09-29 09:31:47,382 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a80519f096ad97fa4f04e1621ce835db) switched from CANCELING to CANCELED. 2020-09-29 09:31:47,383 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution a80519f096ad97fa4f04e1621ce835db. 2020-09-29 09:31:47,383 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (362406cb3731bda85cd6ae975c82c7a1) switched from SCHEDULED to CANCELING. 2020-09-29 09:31:47,383 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (362406cb3731bda85cd6ae975c82c7a1) switched from CANCELING to CANCELED. 2020-09-29 09:31:47,383 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 362406cb3731bda85cd6ae975c82c7a1. 2020-09-29 09:31:47,383 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (506ff1fc18b74615baf0163315a060e1) switched from state FAILING to FAILED. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:31:47,385 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job 506ff1fc18b74615baf0163315a060e1. 2020-09-29 09:31:47,385 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down 2020-09-29 09:31:47,385 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 313e1af898f26a62bf521d6067eb2431. 2020-09-29 09:31:47,386 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{80d221a38fce34a3c528ba7d7d1e9676}] timed out. 2020-09-29 09:31:47,389 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 506ff1fc18b74615baf0163315a060e1 reached globally terminal state FAILED. 2020-09-29 09:31:47,394 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job digestOrchestrator(506ff1fc18b74615baf0163315a060e1). 2020-09-29 09:31:47,397 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending SlotPool. 2020-09-29 09:31:47,397 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection cb5d30f831de374d1dd2bcdcbca1e5bd: JobManager is shutting down.. 2020-09-29 09:31:47,397 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping SlotPool. 2020-09-29 09:31:47,398 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_3 for job 506ff1fc18b74615baf0163315a060e1 from the resource manager. 2020-09-29 09:31:53,494 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 7600c07e0efe65138ee9b76c62febf65 (digestOrchestrator). 2020-09-29 09:31:53,495 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 7600c07e0efe65138ee9b76c62febf65 (digestOrchestrator). 2020-09-29 09:31:53,509 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_4 . 2020-09-29 09:31:53,509 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65). 2020-09-29 09:31:53,526 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) for digestOrchestrator (7600c07e0efe65138ee9b76c62febf65). 2020-09-29 09:31:53,527 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65). 2020-09-29 09:31:53,527 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2020-09-29 09:31:53,527 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 2020-09-29 09:31:53,528 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2020-09-29 09:31:53,528 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using predefined options: DEFAULT. 2020-09-29 09:31:53,528 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}. 2020-09-29 09:31:53,534 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@33507356 for digestOrchestrator (7600c07e0efe65138ee9b76c62febf65). 2020-09-29 09:31:53,534 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@flink-jobmanager:6123/user/rpc/jobmanager_4. 2020-09-29 09:31:53,534 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) under job master id 00000000000000000000000000000000. 2020-09-29 09:31:53,535 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 2020-09-29 09:31:53,535 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state CREATED to RUNNING. 2020-09-29 09:31:53,535 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (05ecea2446c9fb2ab9ad876be6ebef1f) switched from CREATED to SCHEDULED. 2020-09-29 09:31:53,535 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (14ba90c6975372232d94075c4360d8d7) switched from CREATED to SCHEDULED. 2020-09-29 09:31:53,535 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (203cce8e54294d6da5ee76ec8fd8ce4e) switched from CREATED to SCHEDULED. 2020-09-29 09:31:53,536 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{10b08557595fdc3d86e0c421e8fbd8d6}] 2020-09-29 09:31:53,536 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2020-09-29 09:31:53,538 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2020-09-29 09:31:53,538 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_4 for job 7600c07e0efe65138ee9b76c62febf65. 2020-09-29 09:31:53,539 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_4 for job 7600c07e0efe65138ee9b76c62febf65. 2020-09-29 09:31:53,540 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2020-09-29 09:31:53,540 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{10b08557595fdc3d86e0c421e8fbd8d6}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:31:53,540 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 7600c07e0efe65138ee9b76c62febf65 with allocation id 7d22cbce07023734467406b245e92931. 2020-09-29 09:31:57,584 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:31:59,026 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:01,590 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:05,596 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:09,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:09,602 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:13,609 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:17,617 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:19,026 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:21,624 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:32:29,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:39,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:49,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:32:59,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:09,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:19,026 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:29,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:39,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:49,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:33:59,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:09,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:19,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:29,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:39,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:49,026 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:34:59,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:09,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:19,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:29,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:39,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:49,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:35:59,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:09,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:19,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:29,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:39,026 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:49,025 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:36:53,539 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (14ba90c6975372232d94075c4360d8d7) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:36:53,541 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:36:53,541 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:36:53,542 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RUNNING to RESTARTING. 2020-09-29 09:36:53,542 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (05ecea2446c9fb2ab9ad876be6ebef1f) switched from SCHEDULED to CANCELING. 2020-09-29 09:36:53,542 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (05ecea2446c9fb2ab9ad876be6ebef1f) switched from CANCELING to CANCELED. 2020-09-29 09:36:53,543 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 05ecea2446c9fb2ab9ad876be6ebef1f. 2020-09-29 09:36:53,543 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (203cce8e54294d6da5ee76ec8fd8ce4e) switched from SCHEDULED to CANCELING. 2020-09-29 09:36:53,543 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (203cce8e54294d6da5ee76ec8fd8ce4e) switched from CANCELING to CANCELED. 2020-09-29 09:36:53,543 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 203cce8e54294d6da5ee76ec8fd8ce4e. 2020-09-29 09:36:53,544 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 14ba90c6975372232d94075c4360d8d7. 2020-09-29 09:36:53,544 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{10b08557595fdc3d86e0c421e8fbd8d6}] timed out. 2020-09-29 09:37:03,545 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RESTARTING to RUNNING. 2020-09-29 09:37:03,546 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (943c9f99758f2b3cdb7a02de93d7a3eb) switched from CREATED to SCHEDULED. 2020-09-29 09:37:03,546 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (62f61f49bb7bca387c9614e1c94e842e) switched from CREATED to SCHEDULED. 2020-09-29 09:37:03,546 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (de1042ccaa31d276dc1ea9e45d3f5141) switched from CREATED to SCHEDULED. 2020-09-29 09:37:03,548 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{dfe927d6abd70de58588092a9f2ab79a}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:37:03,549 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 7600c07e0efe65138ee9b76c62febf65 with allocation id cf40237809e076831180ef2b8f176002. 2020-09-29 09:37:07,579 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:09,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:37:11,586 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:15,593 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:19,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:37:19,599 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:23,607 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:27,620 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:29,254 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:37:31,627 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:37:39,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:37:49,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:37:59,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:09,254 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:19,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:29,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:39,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:49,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:38:59,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:09,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:19,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:29,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:39,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:49,254 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:39:59,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:09,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:19,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:29,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:39,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:49,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:40:59,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:09,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:19,254 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:29,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:39,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:49,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:41:59,253 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:42:03,554 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (62f61f49bb7bca387c9614e1c94e842e) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:42:03,556 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:42:03,556 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:42:03,557 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RUNNING to RESTARTING. 2020-09-29 09:42:03,557 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (943c9f99758f2b3cdb7a02de93d7a3eb) switched from SCHEDULED to CANCELING. 2020-09-29 09:42:03,557 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (943c9f99758f2b3cdb7a02de93d7a3eb) switched from CANCELING to CANCELED. 2020-09-29 09:42:03,558 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 943c9f99758f2b3cdb7a02de93d7a3eb. 2020-09-29 09:42:03,558 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (de1042ccaa31d276dc1ea9e45d3f5141) switched from SCHEDULED to CANCELING. 2020-09-29 09:42:03,558 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (de1042ccaa31d276dc1ea9e45d3f5141) switched from CANCELING to CANCELED. 2020-09-29 09:42:03,558 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution de1042ccaa31d276dc1ea9e45d3f5141. 2020-09-29 09:42:03,558 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 62f61f49bb7bca387c9614e1c94e842e. 2020-09-29 09:42:03,559 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{dfe927d6abd70de58588092a9f2ab79a}] timed out. 2020-09-29 09:42:13,559 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RESTARTING to RUNNING. 2020-09-29 09:42:13,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (937f95b9049fd584d5a38310afba9878) switched from CREATED to SCHEDULED. 2020-09-29 09:42:13,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (64e0ada940d37ad1f9a40ab8b7201a75) switched from CREATED to SCHEDULED. 2020-09-29 09:42:13,562 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (7f234c5da9ff4e04f78c206c39d3202c) switched from CREATED to SCHEDULED. 2020-09-29 09:42:13,563 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{f0e82f297eddf6c64875ad98d19dedfb}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:42:13,565 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 7600c07e0efe65138ee9b76c62febf65 with allocation id 14204b217d314294a6b5c9c3aefeec6f. 2020-09-29 09:42:17,595 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:19,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:42:21,603 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:25,609 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:29,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:42:29,615 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:33,622 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:37,631 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:39,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:42:41,637 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:42:49,556 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:42:59,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:09,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:19,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:29,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:39,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:49,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:43:59,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:09,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:19,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:29,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:39,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:49,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:44:59,556 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:09,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:19,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:29,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:39,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:49,556 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:45:59,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:09,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:19,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:29,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:39,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:49,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:46:59,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:47:09,555 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:47:13,568 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (64e0ada940d37ad1f9a40ab8b7201a75) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:47:13,570 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:47:13,570 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:47:13,571 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RUNNING to RESTARTING. 2020-09-29 09:47:13,571 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (937f95b9049fd584d5a38310afba9878) switched from SCHEDULED to CANCELING. 2020-09-29 09:47:13,571 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (937f95b9049fd584d5a38310afba9878) switched from CANCELING to CANCELED. 2020-09-29 09:47:13,572 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 937f95b9049fd584d5a38310afba9878. 2020-09-29 09:47:13,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (7f234c5da9ff4e04f78c206c39d3202c) switched from SCHEDULED to CANCELING. 2020-09-29 09:47:13,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (7f234c5da9ff4e04f78c206c39d3202c) switched from CANCELING to CANCELED. 2020-09-29 09:47:13,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 7f234c5da9ff4e04f78c206c39d3202c. 2020-09-29 09:47:13,573 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 64e0ada940d37ad1f9a40ab8b7201a75. 2020-09-29 09:47:13,573 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{f0e82f297eddf6c64875ad98d19dedfb}] timed out. 2020-09-29 09:47:23,574 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RESTARTING to RUNNING. 2020-09-29 09:47:23,576 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8aef2a5ce925e6a35e908ae8875f911e) switched from CREATED to SCHEDULED. 2020-09-29 09:47:23,576 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (e02b71d4534f15e34784dbcc085b89b2) switched from CREATED to SCHEDULED. 2020-09-29 09:47:23,576 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (711d2dcfdd531efedeaf42de6fc3a9bd) switched from CREATED to SCHEDULED. 2020-09-29 09:47:23,577 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{6f4793c1808b997eadd9abe574da3bec}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:47:23,578 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job 7600c07e0efe65138ee9b76c62febf65 with allocation id 8309b99dc63fa85ba5db5b000fe72252. 2020-09-29 09:47:27,605 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:28,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:47:31,612 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:35,618 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:38,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:47:39,624 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:43,630 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:47,636 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:48,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:47:51,647 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.40.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:47:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:08,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:18,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:28,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:38,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:48,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:48:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:08,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:18,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:28,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:38,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:48,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:49:58,896 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:08,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:18,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:28,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:38,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:48,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:50:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:08,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:18,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:28,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:38,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:48,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:51:58,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:08,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:18,895 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job 7600c07e0efe65138ee9b76c62febf65 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:23,581 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (e02b71d4534f15e34784dbcc085b89b2) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:52:23,582 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:52:23,583 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:52:23,583 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state RUNNING to FAILING. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:52:23,587 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8aef2a5ce925e6a35e908ae8875f911e) switched from SCHEDULED to CANCELING. 2020-09-29 09:52:23,587 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8aef2a5ce925e6a35e908ae8875f911e) switched from CANCELING to CANCELED. 2020-09-29 09:52:23,588 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 8aef2a5ce925e6a35e908ae8875f911e. 2020-09-29 09:52:23,588 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (711d2dcfdd531efedeaf42de6fc3a9bd) switched from SCHEDULED to CANCELING. 2020-09-29 09:52:23,588 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (711d2dcfdd531efedeaf42de6fc3a9bd) switched from CANCELING to CANCELED. 2020-09-29 09:52:23,588 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 711d2dcfdd531efedeaf42de6fc3a9bd. 2020-09-29 09:52:23,588 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (7600c07e0efe65138ee9b76c62febf65) switched from state FAILING to FAILED. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:52:23,590 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job 7600c07e0efe65138ee9b76c62febf65. 2020-09-29 09:52:23,590 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down 2020-09-29 09:52:23,590 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution e02b71d4534f15e34784dbcc085b89b2. 2020-09-29 09:52:23,590 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{6f4793c1808b997eadd9abe574da3bec}] timed out. 2020-09-29 09:52:23,593 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 7600c07e0efe65138ee9b76c62febf65 reached globally terminal state FAILED. 2020-09-29 09:52:23,597 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job digestOrchestrator(7600c07e0efe65138ee9b76c62febf65). 2020-09-29 09:52:23,599 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending SlotPool. 2020-09-29 09:52:23,599 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection cb5d30f831de374d1dd2bcdcbca1e5bd: JobManager is shutting down.. 2020-09-29 09:52:23,599 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping SlotPool. 2020-09-29 09:52:23,599 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_4 for job 7600c07e0efe65138ee9b76c62febf65 from the resource manager. 2020-09-29 09:52:29,044 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission f849f4393b17e9b2ec10712e072e2b89 (digestOrchestrator). 2020-09-29 09:52:29,044 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job f849f4393b17e9b2ec10712e072e2b89 (digestOrchestrator). 2020-09-29 09:52:29,059 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_5 . 2020-09-29 09:52:29,059 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89). 2020-09-29 09:52:29,077 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) for digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89). 2020-09-29 09:52:29,078 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89). 2020-09-29 09:52:29,078 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2020-09-29 09:52:29,078 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 2020-09-29 09:52:29,079 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2020-09-29 09:52:29,079 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using predefined options: DEFAULT. 2020-09-29 09:52:29,079 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}. 2020-09-29 09:52:29,085 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@3cdc1e5b for digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89). 2020-09-29 09:52:29,085 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@flink-jobmanager:6123/user/rpc/jobmanager_5. 2020-09-29 09:52:29,086 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) under job master id 00000000000000000000000000000000. 2020-09-29 09:52:29,088 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 2020-09-29 09:52:29,088 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state CREATED to RUNNING. 2020-09-29 09:52:29,090 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8406e332f7eed966606c91ab98a13550) switched from CREATED to SCHEDULED. 2020-09-29 09:52:29,090 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (94c2e150b45693496fe78dc9fff04224) switched from CREATED to SCHEDULED. 2020-09-29 09:52:29,090 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (5eba1f5d83a9fdef7656a7abcb3e2a73) switched from CREATED to SCHEDULED. 2020-09-29 09:52:29,091 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{bbe4fe8a87b1f00aec6d1555d3d8064c}] 2020-09-29 09:52:29,093 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2020-09-29 09:52:29,096 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2020-09-29 09:52:29,097 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_5 for job f849f4393b17e9b2ec10712e072e2b89. 2020-09-29 09:52:29,098 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_5 for job f849f4393b17e9b2ec10712e072e2b89. 2020-09-29 09:52:29,098 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2020-09-29 09:52:29,098 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{bbe4fe8a87b1f00aec6d1555d3d8064c}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:52:29,098 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job f849f4393b17e9b2ec10712e072e2b89 with allocation id f96dc5a07b2592c3d54b1e6671ddb943. 2020-09-29 09:52:33,158 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:36,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:37,164 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:41,171 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:45,177 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:46,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:49,183 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:53,189 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:52:56,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:52:57,196 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:53:06,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:53:16,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:53:26,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:53:36,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:53:46,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:53:56,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:06,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:16,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:26,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:36,463 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:46,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:54:56,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:06,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:16,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:26,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:36,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:46,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:55:56,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:06,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:16,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:26,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:36,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:46,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:56:56,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:06,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:16,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:26,462 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:29,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (94c2e150b45693496fe78dc9fff04224) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 09:57:29,102 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:57:29,103 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 09:57:29,104 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RUNNING to RESTARTING. 2020-09-29 09:57:29,106 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8406e332f7eed966606c91ab98a13550) switched from SCHEDULED to CANCELING. 2020-09-29 09:57:29,106 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (8406e332f7eed966606c91ab98a13550) switched from CANCELING to CANCELED. 2020-09-29 09:57:29,107 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 8406e332f7eed966606c91ab98a13550. 2020-09-29 09:57:29,107 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (5eba1f5d83a9fdef7656a7abcb3e2a73) switched from SCHEDULED to CANCELING. 2020-09-29 09:57:29,107 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (5eba1f5d83a9fdef7656a7abcb3e2a73) switched from CANCELING to CANCELED. 2020-09-29 09:57:29,108 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 5eba1f5d83a9fdef7656a7abcb3e2a73. 2020-09-29 09:57:29,108 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 94c2e150b45693496fe78dc9fff04224. 2020-09-29 09:57:29,108 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{bbe4fe8a87b1f00aec6d1555d3d8064c}] timed out. 2020-09-29 09:57:39,109 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RESTARTING to RUNNING. 2020-09-29 09:57:39,110 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (f645194189a40c1581c2be6f9a0e5508) switched from CREATED to SCHEDULED. 2020-09-29 09:57:39,110 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (5e57bd27899aac65095f8fbf3af390ff) switched from CREATED to SCHEDULED. 2020-09-29 09:57:39,110 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (74249dae97ba72787cbd9c380fcd2547) switched from CREATED to SCHEDULED. 2020-09-29 09:57:39,113 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{4150ddbf45bbc0f9664432726d452c9a}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 09:57:39,114 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job f849f4393b17e9b2ec10712e072e2b89 with allocation id 59ba26104067413d2c5010d2ccc82296. 2020-09-29 09:57:43,147 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:57:47,154 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:57:48,729 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:51,168 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:57:55,176 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:57:58,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:57:59,182 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:58:03,188 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:58:07,195 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 09:58:08,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:58:18,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:58:28,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:58:38,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:58:48,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:58:58,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:08,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:18,728 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:28,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:38,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:48,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 09:59:58,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:08,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:18,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:28,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:38,728 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:48,728 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:00:58,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:08,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:18,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:28,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:38,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:48,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:01:58,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:02:08,728 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:02:18,728 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:02:28,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:02:38,727 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:02:39,117 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (5e57bd27899aac65095f8fbf3af390ff) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:02:39,118 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:02:39,119 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:02:39,119 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RUNNING to RESTARTING. 2020-09-29 10:02:39,120 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (f645194189a40c1581c2be6f9a0e5508) switched from SCHEDULED to CANCELING. 2020-09-29 10:02:39,120 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (f645194189a40c1581c2be6f9a0e5508) switched from CANCELING to CANCELED. 2020-09-29 10:02:39,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution f645194189a40c1581c2be6f9a0e5508. 2020-09-29 10:02:39,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (74249dae97ba72787cbd9c380fcd2547) switched from SCHEDULED to CANCELING. 2020-09-29 10:02:39,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (74249dae97ba72787cbd9c380fcd2547) switched from CANCELING to CANCELED. 2020-09-29 10:02:39,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 74249dae97ba72787cbd9c380fcd2547. 2020-09-29 10:02:39,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 5e57bd27899aac65095f8fbf3af390ff. 2020-09-29 10:02:39,122 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{4150ddbf45bbc0f9664432726d452c9a}] timed out. 2020-09-29 10:02:49,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RESTARTING to RUNNING. 2020-09-29 10:02:49,125 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a01c62d2352e07b8799acd3475585b30) switched from CREATED to SCHEDULED. 2020-09-29 10:02:49,125 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (8a4be765e3ab5d2ffd7a16e5b9625564) switched from CREATED to SCHEDULED. 2020-09-29 10:02:49,125 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (9b72d346a40ab677748c5b4d0406bfaf) switched from CREATED to SCHEDULED. 2020-09-29 10:02:49,125 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{428eb7b8b1279fa33422bf82335e3d6b}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 10:02:49,126 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job f849f4393b17e9b2ec10712e072e2b89 with allocation id f43d8c4bddabacf1e12b437d25b1bdb9. 2020-09-29 10:02:53,153 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:02:57,160 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:02:58,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:01,167 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:03:05,173 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:03:08,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:09,179 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:03:13,185 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:03:17,191 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:03:18,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:28,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:38,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:48,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:03:58,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:08,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:18,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:28,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:38,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:48,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:04:58,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:08,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:18,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:28,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:38,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:48,493 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:05:58,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:08,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:18,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:28,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:38,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:48,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:06:58,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:08,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:18,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:28,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:38,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:48,492 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:07:49,129 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (8a4be765e3ab5d2ffd7a16e5b9625564) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:07:49,130 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:07:49,130 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:07:49,131 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RUNNING to RESTARTING. 2020-09-29 10:07:49,131 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a01c62d2352e07b8799acd3475585b30) switched from SCHEDULED to CANCELING. 2020-09-29 10:07:49,131 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (a01c62d2352e07b8799acd3475585b30) switched from CANCELING to CANCELED. 2020-09-29 10:07:49,132 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution a01c62d2352e07b8799acd3475585b30. 2020-09-29 10:07:49,132 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (9b72d346a40ab677748c5b4d0406bfaf) switched from SCHEDULED to CANCELING. 2020-09-29 10:07:49,132 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (9b72d346a40ab677748c5b4d0406bfaf) switched from CANCELING to CANCELED. 2020-09-29 10:07:49,132 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 9b72d346a40ab677748c5b4d0406bfaf. 2020-09-29 10:07:49,133 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 8a4be765e3ab5d2ffd7a16e5b9625564. 2020-09-29 10:07:49,133 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{428eb7b8b1279fa33422bf82335e3d6b}] timed out. 2020-09-29 10:07:59,133 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RESTARTING to RUNNING. 2020-09-29 10:07:59,135 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (41c1787c60e18309286eea02626c0ed2) switched from CREATED to SCHEDULED. 2020-09-29 10:07:59,135 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (93ec4e1fcfbb3700a17e1ef3b4da2a92) switched from CREATED to SCHEDULED. 2020-09-29 10:07:59,135 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (271786c5460669fdc64d0837642ae099) switched from CREATED to SCHEDULED. 2020-09-29 10:07:59,135 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{e8120c4cce6818e39e2aedc37a4a0882}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 10:07:59,137 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job f849f4393b17e9b2ec10712e072e2b89 with allocation id c658c8df7beb6b1e81401f6479b55c42. 2020-09-29 10:08:03,169 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:05,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:08:07,180 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:11,186 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:15,191 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:15,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:08:19,198 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:23,206 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:25,743 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:08:27,212 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.35.0.3, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:08:35,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:08:45,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:08:55,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:05,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:15,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:25,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:35,743 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:45,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:09:55,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:05,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:15,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:25,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:35,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:45,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:10:55,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:05,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:15,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:25,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:35,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:45,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:11:55,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:05,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:15,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:25,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:35,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:45,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:55,742 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job f849f4393b17e9b2ec10712e072e2b89 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:12:59,140 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (93ec4e1fcfbb3700a17e1ef3b4da2a92) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:12:59,141 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:12:59,142 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:12:59,142 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state RUNNING to FAILING. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:12:59,145 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (41c1787c60e18309286eea02626c0ed2) switched from SCHEDULED to CANCELING. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (41c1787c60e18309286eea02626c0ed2) switched from CANCELING to CANCELED. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 41c1787c60e18309286eea02626c0ed2. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (271786c5460669fdc64d0837642ae099) switched from SCHEDULED to CANCELING. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (271786c5460669fdc64d0837642ae099) switched from CANCELING to CANCELED. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 271786c5460669fdc64d0837642ae099. 2020-09-29 10:12:59,146 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (f849f4393b17e9b2ec10712e072e2b89) switched from state FAILING to FAILED. org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:49) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1710) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1287) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1255) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1086) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.executiongraph.ExecutionVertex.markFailed(ExecutionVertex.java:748) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultExecutionVertexOperations.markFailed(DefaultExecutionVertexOperations.java:41) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskDeploymentFailure(DefaultScheduler.java:435) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] ... 45 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:12:59,147 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job f849f4393b17e9b2ec10712e072e2b89. 2020-09-29 10:12:59,147 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down 2020-09-29 10:12:59,147 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 93ec4e1fcfbb3700a17e1ef3b4da2a92. 2020-09-29 10:12:59,147 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{e8120c4cce6818e39e2aedc37a4a0882}] timed out. 2020-09-29 10:12:59,149 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job f849f4393b17e9b2ec10712e072e2b89 reached globally terminal state FAILED. 2020-09-29 10:12:59,153 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job digestOrchestrator(f849f4393b17e9b2ec10712e072e2b89). 2020-09-29 10:12:59,155 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending SlotPool. 2020-09-29 10:12:59,155 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection cb5d30f831de374d1dd2bcdcbca1e5bd: JobManager is shutting down.. 2020-09-29 10:12:59,155 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping SlotPool. 2020-09-29 10:12:59,155 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_5 for job f849f4393b17e9b2ec10712e072e2b89 from the resource manager. 2020-09-29 10:13:04,653 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission fee646ab106148228144ec615c24367a (digestOrchestrator). 2020-09-29 10:13:04,653 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job fee646ab106148228144ec615c24367a (digestOrchestrator). 2020-09-29 10:13:04,661 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_6 . 2020-09-29 10:13:04,663 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job digestOrchestrator (fee646ab106148228144ec615c24367a). 2020-09-29 10:13:04,684 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=10000) for digestOrchestrator (fee646ab106148228144ec615c24367a). 2020-09-29 10:13:04,685 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job digestOrchestrator (fee646ab106148228144ec615c24367a). 2020-09-29 10:13:04,685 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 2020-09-29 10:13:04,685 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 2020-09-29 10:13:04,686 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2020-09-29 10:13:04,686 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using predefined options: DEFAULT. 2020-09-29 10:13:04,686 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend [] - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}. 2020-09-29 10:13:04,693 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@536bd62c for digestOrchestrator (fee646ab106148228144ec615c24367a). 2020-09-29 10:13:04,693 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job digestOrchestrator (fee646ab106148228144ec615c24367a) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@flink-jobmanager:6123/user/rpc/jobmanager_6. 2020-09-29 10:13:04,693 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job digestOrchestrator (fee646ab106148228144ec615c24367a) under job master id 00000000000000000000000000000000. 2020-09-29 10:13:04,695 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 2020-09-29 10:13:04,695 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (fee646ab106148228144ec615c24367a) switched from state CREATED to RUNNING. 2020-09-29 10:13:04,696 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (927b283d044f5ed25a02fb135ddcd1f8) switched from CREATED to SCHEDULED. 2020-09-29 10:13:04,696 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (b686104099e5bd27666ca57d8f5b5b26) switched from CREATED to SCHEDULED. 2020-09-29 10:13:04,696 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (645c238dde72b66fdb1093498101a40f) switched from CREATED to SCHEDULED. 2020-09-29 10:13:04,696 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{b50aee09b641fbd931d02e4806cddc8d}] 2020-09-29 10:13:04,697 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2020-09-29 10:13:04,700 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2020-09-29 10:13:04,700 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_6 for job fee646ab106148228144ec615c24367a. 2020-09-29 10:13:04,701 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [hidden email]://flink@flink-jobmanager:6123/user/rpc/jobmanager_6 for job fee646ab106148228144ec615c24367a. 2020-09-29 10:13:04,701 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2020-09-29 10:13:04,702 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{b50aee09b641fbd931d02e4806cddc8d}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 10:13:04,702 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job fee646ab106148228144ec615c24367a with allocation id c8c6308ea7e71a942303204ac8dfdf14. 2020-09-29 10:13:08,748 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:10,256 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:13:12,754 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:16,759 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:20,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:13:20,765 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:24,771 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:28,777 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:30,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:13:32,783 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:13:40,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:13:50,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:00,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:10,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:20,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:30,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:40,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:14:50,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:00,256 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:10,256 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:20,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:30,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:40,256 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:15:50,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:00,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:10,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:20,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:30,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:40,256 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:16:50,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:00,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:10,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:20,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:30,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:40,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:17:50,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:18:00,255 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:18:04,702 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (b686104099e5bd27666ca57d8f5b5b26) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:18:04,704 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:18:04,705 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:18:04,705 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (fee646ab106148228144ec615c24367a) switched from state RUNNING to RESTARTING. 2020-09-29 10:18:04,706 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (927b283d044f5ed25a02fb135ddcd1f8) switched from SCHEDULED to CANCELING. 2020-09-29 10:18:04,706 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (927b283d044f5ed25a02fb135ddcd1f8) switched from CANCELING to CANCELED. 2020-09-29 10:18:04,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 927b283d044f5ed25a02fb135ddcd1f8. 2020-09-29 10:18:04,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (645c238dde72b66fdb1093498101a40f) switched from SCHEDULED to CANCELING. 2020-09-29 10:18:04,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (645c238dde72b66fdb1093498101a40f) switched from CANCELING to CANCELED. 2020-09-29 10:18:04,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 645c238dde72b66fdb1093498101a40f. 2020-09-29 10:18:04,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution b686104099e5bd27666ca57d8f5b5b26. 2020-09-29 10:18:04,708 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Pending slot request [SlotRequestId{b50aee09b641fbd931d02e4806cddc8d}] timed out. 2020-09-29 10:18:14,708 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (fee646ab106148228144ec615c24367a) switched from state RESTARTING to RUNNING. 2020-09-29 10:18:14,710 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (45dce2ec5e02a6ab49abb85c0c846f62) switched from CREATED to SCHEDULED. 2020-09-29 10:18:14,710 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (be41da1c9bb400a6b6fa6b39e2b51132) switched from CREATED to SCHEDULED. 2020-09-29 10:18:14,710 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: Output to main-output -> Sink: DIGEST-PROCESSED-MSG) (1/1) (5fd528fc8d2a1894e7b50816765652c9) switched from CREATED to SCHEDULED. 2020-09-29 10:18:14,712 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting new slot [SlotRequestId{59bbdac365fef977afaffa57e8135966}] and profile ResourceProfile{UNKNOWN} from resource manager. 2020-09-29 10:18:14,713 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Request slot with profile ResourceProfile{UNKNOWN} for job fee646ab106148228144ec615c24367a with allocation id 3cc7c136fcce2e2514dab8909db197de. 2020-09-29 10:18:18,755 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:22,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:18:22,762 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:26,768 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:30,776 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:32,678 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:18:34,782 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:38,790 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:42,678 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:18:42,796 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation [] - No hostname could be resolved for the IP address 10.44.0.2, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2020-09-29 10:18:52,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:02,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:12,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:22,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:32,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:42,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:19:52,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:02,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:12,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:22,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:32,678 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:42,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:20:52,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:02,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:12,678 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:22,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:32,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:42,678 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:21:52,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:02,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:12,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:22,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:32,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:42,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:22:52,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:23:02,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:23:12,677 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint triggering task Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) of job fee646ab106148228144ec615c24367a is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2020-09-29 10:23:14,715 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: handlers-response -> Map -> Filter -> Process: Input from Handlers Response (1/1) (be41da1c9bb400a6b6fa6b39e2b51132) switched from SCHEDULED to FAILED on not deployed. org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources. at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?] at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.11.1.jar:1.11.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.11.1.jar:1.11.1] Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] ... 25 more Caused by: java.util.concurrent.TimeoutException ... 23 more 2020-09-29 10:23:14,716 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:23:14,716 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 3 tasks should be restarted to recover the failed task 7c0404ea7e489612a65aff7bbcaf328c_0. 2020-09-29 10:23:14,717 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job digestOrchestrator (fee646ab106148228144ec615c24367a) switched from state RUNNING to RESTARTING. 2020-09-29 10:23:14,717 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (45dce2ec5e02a6ab49abb85c0c846f62) switched from SCHEDULED to CANCELING. 2020-09-29 10:23:14,717 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: DIGEST-INT-MSG -> Map -> Filter -> Process: Input from main-input -> Filter (1/1) (45dce2ec5e02a6ab49abb85c0c846f62) switched from CANCELING to CANCELED. 2020-09-29 10:23:14,718 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 45dce2ec5e02a6ab49abb85c0c846f62. 2020-09-29 10:23:14,718 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process: Orchestrator Job Logic -> (Map -> Process: Output to CELLEBRITE -> Sink: CELLEBRITE-IN, Map -> Process: Output to MAGNET -> Sink: MAGNET-IN, Map -> Process: Output to XRY -> Sink: XRY-IN, Map -> Process: Output to NUIX -> Sink: NUIX-IN, Map -> Process: Output to CYRONIX -> Sink: CYRONIX-IN, Map -> Process: Output to FILES -> Sink: FILES-IN, Map -> Process: Output to IMAGES -> Sink: IMAGES-IN, Map -> Process: Output to ENCASE -> Sink: ENCASE-IN, Map -> Process: Output to PROGRESS_MONITOR -> Sink: PROGRESS-MONITOR-IN, Map -> Process: Output to STOP_JOB -> Map -> Sink: STOPPED-JOBS, Filter: Main output filtered by Stopped Items -> Map -> Process: regards, nick |
|
|
Hi Nick, could you also share the logs of the TaskManagers with us? Moreover, could you set the log level to debug and share the logs again with us? This could help to understand what's going wrong in your case. From the present logs it looks as if the slot request from the ResourceManager to the TaskExecutor or the slot offering from the TaskExecutor to the JobMaster somehow goes wrong/does not work. Cheers, Till On Tue, Sep 29, 2020 at 2:41 PM nick toker <[hidden email]> wrote:
|
|
|
Thanks for sharing the logs with me Nick. Quick question concerning the setup. I see in the logs that multiple JobManagers are being started. Some of them seem to have overlapping logging periods. Is this expected? If you only start a single JobManager, then you should only get the logs of a single instance. One thing I noticed is that the TaskManager only receives the slot request 20 minutes after it has registered at the ResourceManager. Can it be that the cluster is underprovisioned and that the TM pod does not get enough CPU cycles? Cheers, Till On Wed, Sep 30, 2020 at 12:08 PM nick toker <[hidden email]> wrote:
|
|
|
Hi all worked on 1.9.1 and now we moved to 1.11.1 it stop working nick בתאריך יום ד׳, 30 בספט׳ 2020 ב-16:00 מאת Till Rohrmann <[hidden email]>:
|
|
|
Hi Nick, could you maybe run the whole cluster with DEBUG logs enabled? Maybe this helps to pinpoint the problem. Cheers, Till On Thu, Oct 1, 2020 at 4:57 PM nick toker <[hidden email]> wrote:
|
|
|
Hi What do you mean all cluster in debug mode? i open flink to debug in the logs i sent we have only one task and one job manager in the cluster we manger to run it on open system ( to word) but cant run it in a close environment ( that we need) 1. does Flink need to be open to the internet ? 2. what chnaged in resource manager that can effect on close system? regards, nick בתאריך יום ו׳, 2 באוק׳ 2020 ב-12:35 מאת Till Rohrmann <[hidden email]>:
|
|
|
With debug I meant to set the log level of your cluster to DEBUG in the log4j.properties file. Flink does not need to be open wrt the internet. In fact it is highly encouraged to protect your Flink cluster against outside intruders by sealing Flink off using a firewall or other means. Cheers, Till On Sun, Oct 4, 2020 at 8:39 AM nick toker <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |