Fwd: Any Advice on How to build a job cluster in docker container?
Locked
Fwd: Any Advice on How to build a job cluster in docker container?
 
|
Hi, We were following the instructions here https://github.com/apache/flink/tree/release-1.9/flink-container/docker to pack a job and flink together in a single image, but job-cluster/StandaloneJobClusterEntryPoint keeps saying it cannot find the class. Our procedure is like this. ./build.sh --from-archive /Users/muyun/Downloads/flink-1.9.1-bin-scala_2.11.tgz --job-artifacts /usr/local/Cellar/apache-flink/1.9.1/libexec/examples/streaming/Twitter.jar FLINK_JOB=org.apache.flink.streaming.examples.twitter.TwitterExample docker-compose up We were using all the original dockerfile, docker-compose and docker-entry here. The log is pasted below. zhangzongshunde-MacBook-Pro:docker muyun$ ./build.sh --from-archive /Users/muyun/Downloads/flink-1.9.1-bin-scala_2.11.tgz --job-artifacts /usr/local/Cellar/apache-flink/1.9.1/libexec/examples/streaming/Twitter.jar ?????????? --from-archive --job-artifacts 1111111111111 Sending build context to Docker daemon 259.1MB Step 1/23 : FROM openjdk:8-jre-alpine ---> f7a292bbb70c Step 2/23 : RUN apk add --no-cache bash snappy libc6-compat ---> Running in edc6cc04f8aa fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz (1/9) Upgrading musl (1.1.20-r4 -> 1.1.20-r5) (2/9) Installing ncurses-terminfo-base (6.1_p20190105-r0) (3/9) Installing ncurses-terminfo (6.1_p20190105-r0) (4/9) Installing ncurses-libs (6.1_p20190105-r0) (5/9) Installing readline (7.0.003-r1) (6/9) Installing bash (4.4.19-r1) Executing bash-4.4.19-r1.post-install (7/9) Upgrading musl-utils (1.1.20-r4 -> 1.1.20-r5) (8/9) Installing libc6-compat (1.1.20-r5) (9/9) Installing snappy (1.1.7-r1) Executing busybox-1.29.3-r10.trigger OK: 93 MiB in 60 packages Removing intermediate container edc6cc04f8aa ---> ca77b62e27b0 Step 3/23 : ENV FLINK_INSTALL_PATH=/opt ---> Running in c8f5940219ba Removing intermediate container c8f5940219ba ---> cf358a5d963f Step 4/23 : ENV FLINK_HOME $FLINK_INSTALL_PATH/flink ---> Running in 5cffad1a8a57 Removing intermediate container 5cffad1a8a57 ---> 8469271d7ab1 Step 5/23 : ENV FLINK_LIB_DIR $FLINK_HOME/lib ---> Running in d31ef80ec521 Removing intermediate container d31ef80ec521 ---> d3bbdabb4b6d Step 6/23 : ENV FLINK_PLUGINS_DIR $FLINK_HOME/plugins ---> Running in a59f420244be Removing intermediate container a59f420244be ---> af84a0f8f7b4 Step 7/23 : ENV FLINK_OPT_DIR $FLINK_HOME/opt ---> Running in f74c1468f85f Removing intermediate container f74c1468f85f ---> a02b080aa976 Step 8/23 : ENV FLINK_JOB_ARTIFACTS_DIR $FLINK_INSTALL_PATH/artifacts ---> Running in c9683967cc6e Removing intermediate container c9683967cc6e ---> 4b3ff0b1d272 Step 9/23 : ENV FLINK_USR_LIB_DIR $FLINK_HOME/usrlib ---> Running in 4abae2b2d8f3 Removing intermediate container 4abae2b2d8f3 ---> 3094bd6065e2 Step 10/23 : ENV PATH $PATH:$FLINK_HOME/bin ---> Running in 9a24ada189b3 Removing intermediate container 9a24ada189b3 ---> 665ab3dcc0c9 Step 11/23 : ARG flink_dist=NOT_SET ---> Running in 20c70f37dde5 Removing intermediate container 20c70f37dde5 ---> 18a3faba435e Step 12/23 : ARG job_artifacts=NOT_SET ---> Running in c6feb439444e Removing intermediate container c6feb439444e ---> 740d77e8c103 Step 13/23 : ARG python_version=NOT_SET ---> Running in 0b990fe56fbe Removing intermediate container 0b990fe56fbe ---> 3fa6a784370f Step 14/23 : ARG hadoop_jar=NOT_SET* ---> Running in 8c5aa9c58164 Removing intermediate container 8c5aa9c58164 ---> f63568eb5078 Step 15/23 : RUN if [ "$python_version" = "2" ]; then apk add --no-cache python; elif [ "$python_version" = "3" ]; then apk add --no-cache python3 && ln -s /usr/bin/python3 /usr/bin/python; fi ---> Running in 73b4ca9da83e Removing intermediate container 73b4ca9da83e ---> 4fffc60cb175 Step 16/23 : ADD $flink_dist $hadoop_jar $FLINK_INSTALL_PATH/ ---> 45007aa57009 Step 17/23 : ADD $job_artifacts/* $FLINK_JOB_ARTIFACTS_DIR/ ---> 9489209f42e5 Step 18/23 : RUN set -x && ln -s $FLINK_INSTALL_PATH/flink-[0-9]* $FLINK_HOME && ln -s $FLINK_JOB_ARTIFACTS_DIR $FLINK_USR_LIB_DIR && if [ -n "$python_version" ]; then ln -s $FLINK_OPT_DIR/flink-python*.jar $FLINK_LIB_DIR; fi && if [ -f ${FLINK_INSTALL_PATH}/flink-shaded-hadoop* ]; then ln -s ${FLINK_INSTALL_PATH}/flink-shaded-hadoop* $FLINK_LIB_DIR; fi && addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && chown -R flink:flink ${FLINK_INSTALL_PATH}/flink-* && chown -R flink:flink ${FLINK_JOB_ARTIFACTS_DIR}/ && chown -h flink:flink $FLINK_HOME ---> Running in c3857b2ff417 + ln -s /opt/flink-1.9.1 /opt/flink + ln -s /opt/artifacts /opt/flink/usrlib + '[' -n ] + '[' -f '/opt/flink-shaded-hadoop*' ] + addgroup -S flink + adduser -D -S -H -G flink -h /opt/flink flink + chown -R flink:flink /opt/flink-1.9.1 + chown -R flink:flink /opt/artifacts/ + chown -h flink:flink /opt/flink Removing intermediate container c3857b2ff417 ---> d3e23739e1d3 Step 19/23 : COPY docker-entrypoint.sh / ---> 4d9b92bc5f4d Step 20/23 : USER flink ---> Running in d0e0c2a00df7 Removing intermediate container d0e0c2a00df7 ---> ce9ffa12c2b5 Step 21/23 : EXPOSE 8081 6123 ---> Running in accce5845401 Removing intermediate container accce5845401 ---> f207fd2eee27 Step 22/23 : ENTRYPOINT ["/docker-entrypoint.sh"] ---> Running in 423ce5f49f1c Removing intermediate container 423ce5f49f1c ---> b0baa57e67c0 Step 23/23 : CMD ["--help"] ---> Running in dc564ddd6867 Removing intermediate container dc564ddd6867 ---> b6c854805eb9 Successfully built b6c854805eb9 Successfully tagged flink-job:latest zhangzongshunde-MacBook-Pro:docker muyun$ FLINK_JOB=org.apache.flink.streaming.examples.twitter.TwitterExample docker-compose up WARNING: The SAVEPOINT_OPTIONS variable is not set. Defaulting to a blank string. WARNING: The FLINK_JOB_ARGUMENTS variable is not set. Defaulting to a blank string. Creating docker_job-cluster_1 ... done Creating docker_taskmanager_1 ... done Attaching to docker_taskmanager_1, docker_job-cluster_1 job-cluster_1 | !!!!!!!!!!!!!!!! job-cluster_1 | Starting the job-cluster job-cluster_1 | --job-classname org.apache.flink.streaming.examples.twitter.TwitterExample -Djobmanager.rpc.address=job-cluster -Dparallelism.default=1 taskmanager_1 | !!!!!!!!!!!!!!!! taskmanager_1 | Starting the task-manager taskmanager_1 | -Djobmanager.rpc.address=job-cluster taskmanager_1 | Starting taskexecutor as a console application on host b43120bf42dc. job-cluster_1 | Starting standalonejob as a console application on host adaf78f72ef5. taskmanager_1 | 2020-04-09 02:56:11,961 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- taskmanager_1 | 2020-04-09 02:56:11,967 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.9.1, Rev:4d56de8, Date:30.09.2019 @ 11:32:19 CST) taskmanager_1 | 2020-04-09 02:56:11,967 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: flink taskmanager_1 | 2020-04-09 02:56:11,968 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found> job-cluster_1 | 2020-04-09 02:56:11,964 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- job-cluster_1 | 2020-04-09 02:56:11,969 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneJobClusterEntryPoint (Version: 1.9.1, Rev:4d56de8, Date:30.09.2019 @ 11:32:19 CST) job-cluster_1 | 2020-04-09 02:56:11,969 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink taskmanager_1 | 2020-04-09 02:56:11,969 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - IcedTea - 1.8/25.212-b04 taskmanager_1 | 2020-04-09 02:56:11,970 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes job-cluster_1 | 2020-04-09 02:56:11,970 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found> job-cluster_1 | 2020-04-09 02:56:11,971 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - IcedTea - 1.8/25.212-b04 taskmanager_1 | 2020-04-09 02:56:11,971 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /usr/lib/jvm/java-1.8-openjdk/jre job-cluster_1 | 2020-04-09 02:56:11,972 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes job-cluster_1 | 2020-04-09 02:56:11,972 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-1.8-openjdk/jre job-cluster_1 | 2020-04-09 02:56:11,973 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available job-cluster_1 | 2020-04-09 02:56:11,974 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: job-cluster_1 | 2020-04-09 02:56:11,974 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m job-cluster_1 | 2020-04-09 02:56:11,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m job-cluster_1 | 2020-04-09 02:56:11,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink-1.9.1/conf/log4j-console.properties job-cluster_1 | 2020-04-09 02:56:11,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink-1.9.1/conf/logback-console.xml job-cluster_1 | 2020-04-09 02:56:11,975 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: job-cluster_1 | 2020-04-09 02:56:11,976 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir job-cluster_1 | 2020-04-09 02:56:11,976 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink-1.9.1/conf job-cluster_1 | 2020-04-09 02:56:11,976 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --job-classname job-cluster_1 | 2020-04-09 02:56:11,977 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - org.apache.flink.streaming.examples.twitter.TwitterExample job-cluster_1 | 2020-04-09 02:56:11,977 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Djobmanager.rpc.address=job-cluster job-cluster_1 | 2020-04-09 02:56:11,977 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dparallelism.default=1z taskmanager_1 | 2020-04-09 02:56:11,976 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available taskmanager_1 | 2020-04-09 02:56:11,978 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options: job-cluster_1 | 2020-04-09 02:56:11,978 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink-1.9.1/lib/flink-table-blink_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/flink-table_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/log4j-1.2.17.jar:/opt/flink-1.9.1/lib/slf4j-log4j12-1.7.15.jar:/opt/flink-1.9.1/lib/flink-dist_2.11-1.9.1.jar::: job-cluster_1 | 2020-04-09 02:56:11,978 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- taskmanager_1 | 2020-04-09 02:56:11,979 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC taskmanager_1 | 2020-04-09 02:56:11,981 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M taskmanager_1 | 2020-04-09 02:56:11,981 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M taskmanager_1 | 2020-04-09 02:56:11,982 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T taskmanager_1 | 2020-04-09 02:56:11,983 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/opt/flink-1.9.1/conf/log4j-console.properties taskmanager_1 | 2020-04-09 02:56:11,983 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/opt/flink-1.9.1/conf/logback-console.xml taskmanager_1 | 2020-04-09 02:56:11,984 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments: taskmanager_1 | 2020-04-09 02:56:11,984 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Djobmanager.rpc.address=job-cluster taskmanager_1 | 2020-04-09 02:56:11,984 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir taskmanager_1 | 2020-04-09 02:56:11,985 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /opt/flink-1.9.1/conf job-cluster_1 | 2020-04-09 02:56:11,986 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] taskmanager_1 | 2020-04-09 02:56:11,986 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /opt/flink-1.9.1/lib/flink-table-blink_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/flink-table_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/log4j-1.2.17.jar:/opt/flink-1.9.1/lib/slf4j-log4j12-1.7.15.jar:/opt/flink-1.9.1/lib/flink-dist_2.11-1.9.1.jar::: taskmanager_1 | 2020-04-09 02:56:11,988 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- taskmanager_1 | 2020-04-09 02:56:11,999 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT] taskmanager_1 | 2020-04-09 02:56:12,024 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576. taskmanager_1 | 2020-04-09 02:56:12,099 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost taskmanager_1 | 2020-04-09 02:56:12,101 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 taskmanager_1 | 2020-04-09 02:56:12,102 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m taskmanager_1 | 2020-04-09 02:56:12,103 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m taskmanager_1 | 2020-04-09 02:56:12,104 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 taskmanager_1 | 2020-04-09 02:56:12,105 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 taskmanager_1 | 2020-04-09 02:56:12,108 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region job-cluster_1 | 2020-04-09 02:56:12,466 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost job-cluster_1 | 2020-04-09 02:56:12,466 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 job-cluster_1 | 2020-04-09 02:56:12,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m job-cluster_1 | 2020-04-09 02:56:12,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m job-cluster_1 | 2020-04-09 02:56:12,468 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 job-cluster_1 | 2020-04-09 02:56:12,468 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 job-cluster_1 | 2020-04-09 02:56:12,469 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region taskmanager_1 | 2020-04-09 02:56:12,591 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. job-cluster_1 | 2020-04-09 02:56:12,613 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneJobClusterEntryPoint. job-cluster_1 | 2020-04-09 02:56:12,613 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. job-cluster_1 | 2020-04-09 02:56:12,737 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. taskmanager_1 | 2020-04-09 02:56:12,748 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. job-cluster_1 | 2020-04-09 02:56:12,837 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context. job-cluster_1 | 2020-04-09 02:56:12,900 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. taskmanager_1 | 2020-04-09 02:56:12,926 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. job-cluster_1 | 2020-04-09 02:56:12,975 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. job-cluster_1 | 2020-04-09 02:56:12,981 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. job-cluster_1 | 2020-04-09 02:56:13,971 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at job-cluster:6123 taskmanager_1 | 2020-04-09 02:56:14,054 INFO org.apache.flink.configuration.Configuration - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' taskmanager_1 | 2020-04-09 02:56:14,086 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. taskmanager_1 | 2020-04-09 02:56:14,087 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics taskmanager_1 | 2020-04-09 02:56:14,107 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address job-cluster/172.19.0.3:6123. taskmanager_1 | 2020-04-09 02:56:14,524 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address job-cluster/172.19.0.3:6123 taskmanager_1 | 2020-04-09 02:56:14,527 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'b43120bf42dc/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,528 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,530 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,531 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) taskmanager_1 | 2020-04-09 02:56:14,533 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,534 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) taskmanager_1 | 2020-04-09 02:56:14,935 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address job-cluster/172.19.0.3:6123 taskmanager_1 | 2020-04-09 02:56:14,937 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'b43120bf42dc/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,940 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,942 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,943 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) taskmanager_1 | 2020-04-09 02:56:14,945 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:14,947 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) job-cluster_1 | 2020-04-09 02:56:15,701 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started taskmanager_1 | 2020-04-09 02:56:15,748 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address job-cluster/172.19.0.3:6123 taskmanager_1 | 2020-04-09 02:56:15,751 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address 'b43120bf42dc/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:15,753 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:15,755 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:15,756 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) taskmanager_1 | 2020-04-09 02:56:15,758 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.19.0.2': Connection refused (Connection refused) taskmanager_1 | 2020-04-09 02:56:15,759 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed) job-cluster_1 | 2020-04-09 02:56:15,804 INFO akka.remote.Remoting - Starting remoting job-cluster_1 | 2020-04-09 02:56:16,386 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@job-cluster:6123] job-cluster_1 | 2020-04-09 02:56:16,680 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@job-cluster:6123 job-cluster_1 | 2020-04-09 02:56:16,856 INFO org.apache.flink.configuration.Configuration - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' job-cluster_1 | 2020-04-09 02:56:16,909 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-a3776dea-db2e-4eee-a1c3-b888ee6f1c7f job-cluster_1 | 2020-04-09 02:56:16,932 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:33229 - max concurrent requests: 50 - max backlog: 1000 job-cluster_1 | 2020-04-09 02:56:17,004 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. job-cluster_1 | 2020-04-09 02:56:17,017 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at job-cluster:0 job-cluster_1 | 2020-04-09 02:56:17,113 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started job-cluster_1 | 2020-04-09 02:56:17,133 INFO akka.remote.Remoting - Starting remoting job-cluster_1 | 2020-04-09 02:56:17,178 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@job-cluster:36807] job-cluster_1 | 2020-04-09 02:56:17,210 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink-metrics@job-cluster:36807 job-cluster_1 | 2020-04-09 02:56:17,249 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService . taskmanager_1 | 2020-04-09 02:56:17,360 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address job-cluster/172.19.0.3:6123 taskmanager_1 | 2020-04-09 02:56:17,362 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address 'b43120bf42dc' (172.19.0.2) for communication. taskmanager_1 | 2020-04-09 02:56:17,383 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at 172.19.0.2:0 job-cluster_1 | 2020-04-09 02:56:17,460 INFO org.apache.flink.configuration.Configuration - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' job-cluster_1 | 2020-04-09 02:56:17,469 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-60ec0000-2977-48a8-b817-36c0e19cb50e/flink-web-upload does not exist. job-cluster_1 | 2020-04-09 02:56:17,472 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-60ec0000-2977-48a8-b817-36c0e19cb50e/flink-web-upload for file uploads. job-cluster_1 | 2020-04-09 02:56:17,637 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. job-cluster_1 | 2020-04-09 02:56:18,872 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. job-cluster_1 | 2020-04-09 02:56:18,874 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (fallback keys: [{key=jobmanager.web.log.path, isDeprecated=true}])'. taskmanager_1 | 2020-04-09 02:56:19,585 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started job-cluster_1 | 2020-04-09 02:56:19,774 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at job-cluster:8081 job-cluster_1 | 2020-04-09 02:56:19,782 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://job-cluster:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 job-cluster_1 | 2020-04-09 02:56:19,790 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://job-cluster:8081. taskmanager_1 | 2020-04-09 02:56:19,854 INFO akka.remote.Remoting - Starting remoting job-cluster_1 | 2020-04-09 02:56:20,216 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager . job-cluster_1 | 2020-04-09 02:56:20,261 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Shutting down rest endpoint. job-cluster_1 | 2020-04-09 02:56:20,409 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Removing cache directory /tmp/flink-web-60ec0000-2977-48a8-b817-36c0e19cb50e/flink-web-ui job-cluster_1 | 2020-04-09 02:56:20,448 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://job-cluster:8081 lost leadership job-cluster_1 | 2020-04-09 02:56:20,449 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Shut down complete. job-cluster_1 | 2020-04-09 02:56:20,479 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Shutting StandaloneJobClusterEntryPoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent. job-cluster_1 | at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) job-cluster_1 | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) job-cluster_1 | at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:110) job-cluster_1 | Caused by: org.apache.flink.util.FlinkException: Could not load the provided entrypoint class. job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:119) job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96) job-cluster_1 | at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62) job-cluster_1 | at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41) job-cluster_1 | at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184) job-cluster_1 | ... 6 more job-cluster_1 | Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.examples.twitter.TwitterExample job-cluster_1 | at java.net.URLClassLoader.findClass(URLClassLoader.java:382) job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424) job-cluster_1 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357) job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:116) job-cluster_1 | ... 10 more job-cluster_1 | . job-cluster_1 | 2020-04-09 02:56:20,502 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:33229 job-cluster_1 | 2020-04-09 02:56:20,511 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. job-cluster_1 | 2020-04-09 02:56:20,557 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. taskmanager_1 | 2020-04-09 02:56:20,665 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@172.19.0.2:42409] job-cluster_1 | 2020-04-09 02:56:20,668 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. job-cluster_1 | 2020-04-09 02:56:20,686 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. job-cluster_1 | 2020-04-09 02:56:20,800 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. job-cluster_1 | 2020-04-09 02:56:20,811 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. job-cluster_1 | 2020-04-09 02:56:20,936 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. job-cluster_1 | 2020-04-09 02:56:20,972 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. job-cluster_1 | 2020-04-09 02:56:21,070 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. job-cluster_1 | 2020-04-09 02:56:21,100 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not start cluster entrypoint StandaloneJobClusterEntryPoint. job-cluster_1 | org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneJobClusterEntryPoint. job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) job-cluster_1 | at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:110) job-cluster_1 | Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent. job-cluster_1 | at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) job-cluster_1 | at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) job-cluster_1 | at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) job-cluster_1 | ... 2 more job-cluster_1 | Caused by: org.apache.flink.util.FlinkException: Could not load the provided entrypoint class. job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:119) job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96) job-cluster_1 | at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62) job-cluster_1 | at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41) job-cluster_1 | at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184) job-cluster_1 | ... 6 more job-cluster_1 | Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.examples.twitter.TwitterExample job-cluster_1 | at java.net.URLClassLoader.findClass(URLClassLoader.java:382) job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424) job-cluster_1 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357) job-cluster_1 | at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:116) job-cluster_1 | ... 10 more job-cluster_1 | 2020-04-09 02:56:21,109 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. taskmanager_1 | 2020-04-09 02:56:21,366 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@172.19.0.2:42409 taskmanager_1 | 2020-04-09 02:56:21,467 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. taskmanager_1 | 2020-04-09 02:56:21,490 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at 172.19.0.2:0 docker_job-cluster_1 exited with code 1 taskmanager_1 | 2020-04-09 02:56:21,652 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started taskmanager_1 | 2020-04-09 02:56:21,663 INFO akka.remote.Remoting - Starting remoting taskmanager_1 | 2020-04-09 02:56:21,700 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.19.0.2:46221] taskmanager_1 | 2020-04-09 02:56:21,730 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink-metrics@172.19.0.2:46221 taskmanager_1 | 2020-04-09 02:56:21,795 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService_42842b0bdde3f9aaeb0b7f47e5e71985 . taskmanager_1 | 2020-04-09 02:56:21,942 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-a9e0e79d-d077-4b0e-b126-db1dc1f50e42 taskmanager_1 | 2020-04-09 02:56:21,971 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-247a9740-77f3-4ba7-abde-dbcc067e5896 taskmanager_1 | 2020-04-09 02:56:21,984 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager with ResourceID: 42842b0bdde3f9aaeb0b7f47e5e71985 taskmanager_1 | 2020-04-09 02:56:22,485 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 58 GB, usable 53 GB (91.38% usable) taskmanager_1 | 2020-04-09 02:56:22,499 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager uses directory /tmp/flink-io-050faec7-167c-473c-9359-4388f9d9473b for spill files. taskmanager_1 | 2020-04-09 02:56:22,541 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: /172.19.0.2, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 1 (manual), number of client threads: 1 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)] taskmanager_1 | 2020-04-09 02:56:22,552 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager uses directory /tmp/flink-netty-shuffle-84d43078-9ba1-4e33-86d0-9058c8364889 for spill files. taskmanager_1 | 2020-04-09 02:56:22,947 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768). taskmanager_1 | 2020-04-09 02:56:22,966 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Starting the network environment and its components. taskmanager_1 | 2020-04-09 02:56:23,182 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 214 ms). taskmanager_1 | 2020-04-09 02:56:23,413 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 223 ms). Listening on SocketAddress /172.19.0.2:43695. taskmanager_1 | 2020-04-09 02:56:23,418 INFO org.apache.flink.runtime.taskexecutor.KvStateService - Starting the kvState service and its components. taskmanager_1 | 2020-04-09 02:56:23,419 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (641 MB), memory will be allocated lazily. taskmanager_1 | 2020-04-09 02:56:23,472 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms taskmanager_1 | 2020-04-09 02:56:23,507 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 . taskmanager_1 | 2020-04-09 02:56:23,576 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service. taskmanager_1 | 2020-04-09 02:56:23,582 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-d608e54b-67f8-42cb-bb82-94ab49573258 taskmanager_1 | 2020-04-09 02:56:23,597 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@job-cluster:6123/user/resourcemanager(00000000000000000000000000000000). taskmanager_1 | 2020-04-09 02:56:33,655 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@job-cluster:6123/), Path(/user/resourcemanager)]] after [10000 ms]. Message of type [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.. taskmanager_1 | 2020-04-09 02:56:43,849 INFO akka.remote.transport.ProtocolStateActor - No response from remote for outbound association. Associate timed out after [20000 ms]. taskmanager_1 | 2020-04-09 02:56:43,868 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [No response from remote for outbound association. Associate timed out after [20000 ms].] taskmanager_1 | 2020-04-09 02:56:43,886 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:56:43,999 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with org.apache.flink.shaded.akka.org.jboss.netty.channel.ConnectTimeoutException: connection timed out: job-cluster/172.19.0.3:6123 taskmanager_1 | 2020-04-09 02:56:53,912 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:56:53,914 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:03,901 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster] taskmanager_1 | 2020-04-09 02:57:03,903 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:13,924 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:57:13,925 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:23,955 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:57:23,958 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:33,947 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster] taskmanager_1 | 2020-04-09 02:57:33,948 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:43,973 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:57:43,975 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:57:54,001 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:57:54,002 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:03,990 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster] taskmanager_1 | 2020-04-09 02:58:03,992 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:14,017 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:14,019 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:58:24,048 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:58:24,053 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:34,040 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster] taskmanager_1 | 2020-04-09 02:58:34,040 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:44,066 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:58:44,067 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:58:54,094 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:58:54,097 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:59:04,113 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:59:04,113 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:59:14,162 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:59:14,169 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager.. taskmanager_1 | 2020-04-09 02:59:24,228 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@job-cluster:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@job-cluster:6123]] Caused by: [java.net.UnknownHostException: job-cluster: Name does not resolve] taskmanager_1 | 2020-04-09 02:59:24,251 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@job-cluster:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@job-cluster:6123/user/resourcemanager..
We also tried to start from here https://github.com/apache/flink-docker/tree/346809e6cab2ca0ac702fb4d2bf56afb6ee3c437/1.10/scala_2.12-debian. But that causes another issue that we we want to setup jobmanager and submit job at the same time which if we do something like command: > with which, we think we will be in a race condition. This turns out that we would need to use "$FLINK_HOME/bin/standalone-job.sh --job-classname" to start the jobmanager with the job class at the same time. But this results in the same error above again. In this case we only changed the line here https://github.com/apache/flink-docker/blob/346809e6cab2ca0ac702fb4d2bf56afb6ee3c437/1.10/scala_2.12-debian/docker-entrypoint.sh#L91 to exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "$@" And then we run the docker compose in bash,
FLINK_JOB=org.apache.flink.streaming.examples.twitter.TwitterExample docker-compose up
Thank you so much! Any help is appreciated! Stay healthy. Thanks, Zongshun Zhang |
|
|
It seems that the artifacts are not located in the flink lib directory. You could use "docker run" your image to check the existence. Best, Yang Zhang Zongshun <[hidden email]> 于2020年4月9日周四 上午11:32写道：
|
Re: Any Advice on How to build a job cluster in docker container?
|
|
Hi Yang, Are you saying that all the artifacts should be put inside /opt/flink-1.9.1/lib/? /opt/flink-1.9.1/usrlib In the job-cluster log there is a line job-cluster_1 | 2020-04-09 02:56:11,978 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink-1.9.1/lib/flink-table-blink_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/flink-table_2.11-1.9.1.jar:/opt/flink-1.9.1/lib/log4j-1.2.17.jar:/opt/flink-1.9.1/lib/slf4j-log4j12-1.7.15.jar:/opt/flink-1.9.1/lib/flink-dist_2.11-1.9.1.jar::: Do you know how to include it? Thank you so much! Stay healthy! Best, Zongshun Zhang On Fri, Apr 10, 2020 at 4:37 AM Yang Wang <[hidden email]> wrote:
|
Re: Any Advice on How to build a job cluster in docker container?
|
|
I mean put my tar to /opt/flink-1.9.1/usrlib does work. But it doesn't seem to be the common way for flink to detect my tar. Do you know how to enable flink to scan artifacts in a usrlib? Thanks, Zongshun Zhang On Tue, Apr 14, 2020 at 1:23 PM Zhang Zongshun <[hidden email]> wrote:
|
|
|
Hi Zongshun, Before 1.10, when building the docker images the user artifacts will be copied to the $FLINK_HOME/lib, so they are always in the classpath. However, the behavior changed in 1.10, the urslib has been introduced[1]. I guess you are using the Dockerfile with 1.10 and flink-dist with 1.9. Could you check for that? Best, Yang Zhang Zongshun <[hidden email]> 于2020年4月15日周三 上午3:17写道：
|
| Free forum by Nabble | Edit this page |