Install 1.7.2 on EC2 - No task slots - 2019

[Jeff Crane]

When downloading the latest 1.7.2 and extracting to (a free) AMZ EC2, the daemon (./bin/start-cluster.sh) reports 0 task managers, 0 task slots, and 0 avails, out of the box. All jobs fail. All ports are open to all traffic. 

Can anyone tell me what I missed?

[Konstantin Knauf-2]

Hi Jeff,

do you see any log files in the log directory of your Flink installation directory? If so, please share them.

Cheers,

Konstantin

On Sat, Mar 23, 2019 at 5:36 PM Jeff Crane <[hidden email]> wrote:

When downloading the latest 1.7.2 and extracting to (a free) AMZ EC2, the daemon (./bin/start-cluster.sh) reports 0 task managers, 0 task slots, and 0 avails, out of the box. All jobs fail. All ports are open to all traffic. 

Can anyone tell me what I missed?

--

[Jeff Crane]

There are 2 out files and a log file. I will describe the steps to take below, but I have not found anything to indicate there's a problem when it cannot allocate any resources (but otherwise runs).

----------

start default free tier aws instance

edit security group to allow 8081 incoming

sudo yum install java

# wget http://ftp.naz.com/apache/flink/flink-1.7.2/flink-1.7.2-bin-scala_2.12.tgz

# tar xzvf flink-1.7.2-bin-scala_2.12.tgz

# cd flink-1.7.2/

# ./bin/start-cluster.sh

Web interface shows 0,0,0 for resources.

While there are 2 other .out files, I dunno if those are important.

Here is the log of the only .log -

flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log

2019-03-26 15:43:12,488 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------

2019-03-26 15:43:12,489 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting StandaloneSessionClusterEntrypoint (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC)

2019-03-26 15:43:12,489 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: ec2-user

2019-03-26 15:43:12,489 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: <no hadoop dependency found>

2019-03-26 15:43:12,489 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.191-b12

2019-03-26 15:43:12,490 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 989 MiBytes

2019-03-26 15:43:12,490 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: (not set)

2019-03-26 15:43:12,490 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  No Hadoop Dependency available

2019-03-26 15:43:12,490 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:

2019-03-26 15:43:12,490 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms1024m

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx1024m

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:/home/ec2-user/flink-1.7.2/conf/log4j.properties

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlogback.configurationFile=file:/home/ec2-user/flink-1.7.2/conf/logback.xml

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments:

2019-03-26 15:43:12,491 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     --configDir

2019-03-26 15:43:12,492 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     /home/ec2-user/flink-1.7.2/conf

2019-03-26 15:43:12,492 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     --executionMode

2019-03-26 15:43:12,492 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     cluster

2019-03-26 15:43:12,492 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: /home/ec2-user/flink-1.7.2/lib/flink-python_2.12-1.7.2.jar:/home/ec2-user/flink-1.7.2/lib/log4j-1.2.17.jar:/home/ec2-user/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/home/ec2-user/flink-1.7.2/lib/flink-dist_2.12-1.7.2.jar:::

2019-03-26 15:43:12,492 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------

2019-03-26 15:43:12,495 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]

2019-03-26 15:43:12,531 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost

2019-03-26 15:43:12,533 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123

2019-03-26 15:43:12,533 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m

2019-03-26 15:43:12,534 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m

2019-03-26 15:43:12,534 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1

2019-03-26 15:43:12,534 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1

2019-03-26 15:43:12,535 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081

2019-03-26 15:43:12,705 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting StandaloneSessionClusterEntrypoint.

2019-03-26 15:43:12,705 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.

2019-03-26 15:43:12,714 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.

2019-03-26 15:43:12,730 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install security context.

2019-03-26 15:43:12,742 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.

2019-03-26 15:43:12,761 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.

2019-03-26 15:43:12,762 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.

2019-03-26 15:43:13,432 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at localhost:6123

2019-03-26 15:43:14,281 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started

2019-03-26 15:43:14,418 INFO  akka.remote.Remoting                                          - Starting remoting

2019-03-26 15:43:14,669 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@localhost:6123]

2019-03-26 15:43:14,692 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@localhost:6123

2019-03-26 15:43:14,711 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'

2019-03-26 15:43:14,724 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-cbac7ece-9cb3-4bf8-a225-26fe4136067d

2019-03-26 15:43:14,728 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:45861 - max concurrent requests: 50 - max backlog: 1000

2019-03-26 15:43:14,760 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.

2019-03-26 15:43:14,762 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Trying to start actor system at localhost:0

2019-03-26 15:43:14,813 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started

2019-03-26 15:43:14,829 INFO  akka.remote.Remoting                                          - Starting remoting

2019-03-26 15:43:14,849 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Actor system started at akka.tcp://flink-metrics@localhost:38181

2019-03-26 15:43:14,853 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@localhost:38181]

2019-03-26 15:43:14,861 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-99cbce6e-06d9-4159-b68e-e461b31e41cf, expiration time 3600000, maximum cache size 52428800 bytes.

2019-03-26 15:43:14,898 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /tmp/blobStore-d8d4eb58-74fd-4f6f-9b9b-e71a49228652

2019-03-26 15:43:14,915 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'

2019-03-26 15:43:14,917 WARN  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-027518bf-4744-4f65-9bc0-c0d8e23853b0/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.

2019-03-26 15:43:14,918 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-027518bf-4744-4f65-9bc0-c0d8e23853b0/flink-web-upload for file uploads.

2019-03-26 15:43:14,926 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.

2019-03-26 15:43:15,286 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log

2019-03-26 15:43:15,286 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.out

2019-03-26 15:43:15,504 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at localhost:8081

2019-03-26 15:43:15,504 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - http://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000

2019-03-26 15:43:15,504 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://localhost:8081.

2019-03-26 15:43:15,670 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .

2019-03-26 15:43:15,713 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .

2019-03-26 15:43:15,744 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000

2019-03-26 15:43:15,745 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.

2019-03-26 15:43:15,781 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000

2019-03-26 15:43:15,783 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.

----------

On Monday, March 25, 2019, 1:23:45 AM PDT, Konstantin Knauf <[hidden email]> wrote:

Hi Jeff, 

do you see any log files in the log directory of your Flink installation directory? If so, please share them. 

Cheers, 

Konstantin