(DEPRECATED) Apache Flink User Mailing List archive. - Re: How to start a TaskManager and the Flink Query Interface from, Scala IDE (local run configuration)

(DEPRECATED) Apache Flink User Mailing List archive.

Re: How to start a TaskManager and the Flink Query Interface from, Scala IDE (local run configuration)

Posted by rmetzger0 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Re-How-to-start-a-TaskManager-and-the-Flink-Query-Interface-from-Scala-IDE-local-run-configuration-tp815p816.html

Hi,

this looks actually quite good.

Is the JobManager web interface accessible on port 8081 ?

With the exec mode set " --executionMode LOCAL" there is no explicit start of a TaskManager required. Its running in the same JVM as the JobManager.

The Flink Query Interface needs to be started using another main class: org.apache.flink.client.WebFrontend.

Let me know if you need more help!

Robert

On Sat, Mar 7, 2015 at 6:08 PM, Mihail Vieru <[hidden email]> wrote:

Hey Robert,

the logging level was INFO and I didn't get any errors.
After I switched to DEBUG I get the following output:

2015-03-07 17:46:52 DEBUG MutableMetricsFactory:42 - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
2015-03-07 17:46:52 DEBUG MutableMetricsFactory:42 - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
2015-03-07 17:46:52 DEBUG MetricsSystemImpl:220 - UgiMetrics, User and group related metrics
2015-03-07 17:46:53 DEBUG KerberosName:87 - Kerberos krb5 configuration not found, setting default realm to empty
2015-03-07 17:46:53 DEBUG Groups:180 - Creating new Groups object
2015-03-07 17:46:53 DEBUG NativeCodeLoader:46 - Trying to load the custom-built native-hadoop library...
2015-03-07 17:46:53 DEBUG NativeCodeLoader:55 - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2015-03-07 17:46:53 DEBUG NativeCodeLoader:56 - java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-03-07 17:46:53 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-03-07 17:46:53 DEBUG JniBasedUnixGroupsMappingWithFallback:40 - Falling back to shell based
2015-03-07 17:46:53 DEBUG JniBasedUnixGroupsMappingWithFallback:44 - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2015-03-07 17:46:53 DEBUG Groups:66 - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
2015-03-07 17:46:53 DEBUG UserGroupInformation:177 - hadoop login
2015-03-07 17:46:53 DEBUG UserGroupInformation:126 - hadoop login commit
2015-03-07 17:46:53 DEBUG UserGroupInformation:156 - using local user:UnixPrincipal: vieru
2015-03-07 17:46:53 DEBUG UserGroupInformation:703 - UGI loginUser:vieru (auth:SIMPLE)
2015-03-07 17:46:53 INFO JobManager:225 - --------------------------------------------------------------------------------
2015-03-07 17:46:53 INFO JobManager:226 - Starting JobManager (Version: <unknown>, Rev:84c11e6, Date:<a href="tel:02.03.2015" value="+492032015" target="_blank">02.03.2015 @ 14:53:15 CET)
2015-03-07 17:46:53 INFO JobManager:228 - Current user: vieru
2015-03-07 17:46:53 INFO JobManager:229 - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.7/24.60-b09
2015-03-07 17:46:53 INFO JobManager:235 - Startup Options:
2015-03-07 17:46:53 INFO JobManager:237 - -Dlog4j.configuration=file:///home/vieru/dev/flink-from-src/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/conf/log4j.properties3
2015-03-07 17:46:53 INFO JobManager:237 - -Dfile.encoding=UTF-8
2015-03-07 17:46:53 INFO JobManager:237 - -Xbootclasspath/p:/home/vieru/dev/scala-ide/configuration/org.eclipse.osgi/bundles/290/1/.cp/lib/scala-library.jar:/home/vieru/dev/scala-ide/configuration/org.eclipse.osgi/bundles/290/1/.cp/lib/scala-swing.jar:/home/vieru/dev/scala-ide/configuration/org.eclipse.osgi/bundles/290/1/.cp/lib/scala-actors.jar:/home/vieru/dev/scala-ide/configuration/org.eclipse.osgi/bundles/289/1/.cp/lib/scala-reflect.jar
2015-03-07 17:46:53 INFO JobManager:241 - Maximum heap size: <a href="tel:1682" value="+491682" target="_blank">1682 MiBytes
2015-03-07 17:46:53 INFO JobManager:242 - JAVA_HOME: not set
2015-03-07 17:46:53 INFO JobManager:243 - --------------------------------------------------------------------------------
2015-03-07 17:46:53 INFO JobManager:774 - Loading configuration from /home/vieru/dev/flink-from-src/flink/flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT/conf
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: jobmanager.rpc.address, localhost
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: jobmanager.rpc.port, 6123
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: jobmanager.heap.mb, 256
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: taskmanager.heap.mb, 512
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: taskmanager.numberOfTaskSlots, -1
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: parallelization.degree.default, 1
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: jobmanager.web.port, 8081
2015-03-07 17:46:53 DEBUG GlobalConfiguration:254 - Loading configuration property: webclient.port, 8080
2015-03-07 17:46:53 INFO JobManager:650 - Security is not enabled. Starting non-authenticated JobManager.
2015-03-07 17:46:53 INFO JobManager:681 - Starting JobManager
2015-03-07 17:46:53 INFO JobManager:684 - Starting JobManager actor system at localhost:6123
2015-03-07 17:46:53 DEBUG JobManager:690 - Using akka configuration
Config(SimpleConfigObject({"akka":{"actor":{"default-dispatcher":{"fork-join-executor":{"parallelism-factor":2},"throughput":15},"guardian-supervisor-strategy":"akka.actor.StoppingSupervisorStrategy","provider":"akka.remote.RemoteActorRefProvider"},"daemonic":"on","jvm-exit-on-fatal-error":"off","log-config-on-start":"off","log-dead-letters":"off","log-dead-letters-during-shutdown":"off","loggers":["akka.event.slf4j.Slf4jLogger"],"logging-filter":"akka.event.slf4j.Slf4jLoggingFilter","loglevel":"DEBUG","remote":{"gate-invalid-addresses-for":"5 s","log-remote-lifecycle-events":"off","netty":{"tcp":{"connection-timeout":"100 seconds","hostname":"127.0.0.1","maximum-frame-size":"10485760b","port":6123,"tcp-nodelay":"on","transport-class":"akka.remote.transport.netty.NettyTransport"}},"quarantine-systems-for":"off","startup-timeout":"100 seconds","transport-failure-detector":{"acceptable-heartbeat-pause":"6000 s","heartbeat-interval":"1000 s","threshold":300},"watch-failure-detector":{"acceptable-heartbeat-pause":"100 seconds","heartbeat-interval":"10 seconds","threshold":12}},"serialize-messages":"off","stdout-loglevel":"OFF"}}))
2015-03-07 17:46:53 INFO Slf4jLogger:80 - Slf4jLogger started
2015-03-07 17:46:53 DEBUG EventStream:77 - logger log1-Slf4jLogger started
2015-03-07 17:46:53 DEBUG EventStream:77 - Default Loggers started
2015-03-07 17:46:54 INFO Remoting:74 - Starting remoting
2015-03-07 17:46:54 INFO Remoting:74 - Remoting started; listening on addresses :[akka.tcp://flink@127.0.0.1:6123]
2015-03-07 17:46:54 INFO JobManager:710 - Starting JobManager actor
2015-03-07 17:46:54 INFO BlobServer:83 - Created BLOB server storage directory /tmp/blobStore-3ff4969b-e9a1-4ba7-bae9-1c3896b8bc90
2015-03-07 17:46:54 INFO BlobServer:122 - Started BLOB server at 0.0.0.0:59575 - max concurrent requests: 50 - max backlog: 1000
2015-03-07 17:46:54 DEBUG JobManager:715 - Starting JobManager process reaper
2015-03-07 17:46:54 INFO JobManager:729 - Starting JobManger web frontend
2015-03-07 17:46:54 INFO JobManager:110 - Starting JobManager at akka://flink/user/jobmanager.
2015-03-07 17:46:54 INFO WebInfoServer:113 - Setting up web info server, using web-root directory file:/home/vieru/dev/flink-from-src/flink/flink-runtime/target/classes/web-docs-infoserver.
2015-03-07 17:46:54 DEBUG log:70 - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
2015-03-07 17:46:54 DEBUG log:70 - Container org.eclipse.jetty.server.Server@16f376a3 + SelectChannelConnector@0.0.0.0:8081 as connector
2015-03-07 17:46:54 DEBUG log:70 - filterNameMap={}
2015-03-07 17:46:54 DEBUG log:70 - pathFilters=null
2015-03-07 17:46:54 DEBUG log:70 - servletFilterMap=null
2015-03-07 17:46:54 DEBUG log:70 - servletPathMap={/jobsInfo=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007}
2015-03-07 17:46:54 DEBUG log:70 - servletNameMap={org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007}
2015-03-07 17:46:54 DEBUG log:70 - filterNameMap={}
2015-03-07 17:46:54 DEBUG log:70 - pathFilters=null
2015-03-07 17:46:54 DEBUG log:70 - servletFilterMap=null
2015-03-07 17:46:54 DEBUG log:70 - servletPathMap={/jobsInfo=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, /logInfo=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298}
2015-03-07 17:46:54 DEBUG log:70 - servletNameMap={org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298}
2015-03-07 17:46:54 DEBUG log:70 - filterNameMap={}
2015-03-07 17:46:54 DEBUG log:70 - pathFilters=null
2015-03-07 17:46:54 DEBUG log:70 - servletFilterMap=null
2015-03-07 17:46:54 DEBUG log:70 - servletPathMap={/jobsInfo=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, /setupInfo=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137, /logInfo=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298}
2015-03-07 17:46:54 DEBUG log:70 - servletNameMap={org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298, org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137}
2015-03-07 17:46:54 DEBUG log:70 - filterNameMap={}
2015-03-07 17:46:54 DEBUG log:70 - pathFilters=null
2015-03-07 17:46:54 DEBUG log:70 - servletFilterMap=null
2015-03-07 17:46:54 DEBUG log:70 - servletPathMap={/jobsInfo=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, /setupInfo=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137, /logInfo=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298, /menu=org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192}
2015-03-07 17:46:54 DEBUG log:70 - servletNameMap={org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192=org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192, org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298, org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137}
2015-03-07 17:46:54 DEBUG log:70 - Container org.eclipse.jetty.server.Server@16f376a3 + HandlerList@50dafd61 as handler
2015-03-07 17:46:54 DEBUG log:70 - Container HandlerList@50dafd61 + ServletContextHandler@593114bf@593114bf/,null as handler
2015-03-07 17:46:54 DEBUG log:70 - Container HandlerList@50dafd61 + ResourceHandler@25493558 as handler
2015-03-07 17:46:54 DEBUG log:70 - Starting org.eclipse.jetty.server.Server@16f376a3
2015-03-07 17:46:54 INFO log:55 - jetty-8.0.0.M1
2015-03-07 17:46:54 DEBUG log:70 - Container org.eclipse.jetty.server.Server@16f376a3 + qtp176836489{8<=0<=0/254,-1} as threadpool
2015-03-07 17:46:54 DEBUG log:70 - Starting qtp176836489{8<=0<=0/254,-1}
2015-03-07 17:46:54 DEBUG log:70 - STARTED qtp176836489{8<=7<=8/254,0}
2015-03-07 17:46:54 DEBUG log:70 - Starting HandlerList@50dafd61
2015-03-07 17:46:54 DEBUG log:70 - Starting ServletContextHandler@593114bf@593114bf/,null
2015-03-07 17:46:54 DEBUG log:70 - Container ServletContextHandler@593114bf@593114bf/,null + SessionHandler@679a7ae9@ as handler
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007 as servlet
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298 as servlet
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137 as servlet
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192 as servlet
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + [/jobsInfo]=>org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007 as servletMapping
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + [/logInfo]=>org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298 as servletMapping
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + [/setupInfo]=>org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137 as servletMapping
2015-03-07 17:46:54 DEBUG log:70 - Container ServletHandler@4efb8c22@ + [/menu]=>org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192 as servletMapping
2015-03-07 17:46:54 DEBUG log:70 - Container SessionHandler@679a7ae9@ + ServletHandler@4efb8c22@ as handler
2015-03-07 17:46:54 DEBUG log:70 - Container SessionHandler@679a7ae9@ + org.eclipse.jetty.server.session.HashSessionManager@64ad34b0 as sessionManager
2015-03-07 17:46:54 DEBUG log:70 - Starting SessionHandler@679a7ae9@
2015-03-07 17:46:54 DEBUG log:70 - Starting org.eclipse.jetty.server.session.HashSessionManager@64ad34b0
2015-03-07 17:46:54 DEBUG log:70 - Container org.eclipse.jetty.server.Server@16f376a3 + org.eclipse.jetty.server.session.HashSessionIdManager@a3f075 as sessionIdManager
2015-03-07 17:46:54 DEBUG log:70 - Starting org.eclipse.jetty.server.session.HashSessionIdManager@a3f075
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.eclipse.jetty.server.session.HashSessionIdManager@a3f075
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.eclipse.jetty.server.session.HashSessionManager@64ad34b0
2015-03-07 17:46:54 DEBUG log:70 - Starting ServletHandler@4efb8c22@
2015-03-07 17:46:54 DEBUG log:70 - filterNameMap={}
2015-03-07 17:46:54 DEBUG log:70 - pathFilters=null
2015-03-07 17:46:54 DEBUG log:70 - servletFilterMap=null
2015-03-07 17:46:54 DEBUG log:70 - servletPathMap={/jobsInfo=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, /setupInfo=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137, /logInfo=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298, /menu=org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192}
2015-03-07 17:46:54 DEBUG log:70 - servletNameMap={org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192=org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192, org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007=org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007, org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298=org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298, org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137=org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137}
2015-03-07 17:46:54 DEBUG log:70 - starting ServletHandler@4efb8c22@
2015-03-07 17:46:54 DEBUG log:70 - STARTED ServletHandler@4efb8c22@
2015-03-07 17:46:54 DEBUG log:70 - starting SessionHandler@679a7ae9@
2015-03-07 17:46:54 DEBUG log:70 - STARTED SessionHandler@679a7ae9@
2015-03-07 17:46:54 DEBUG log:70 - starting ServletContextHandler@593114bf@593114bf/,null
2015-03-07 17:46:54 DEBUG log:70 - Starting org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007
2015-03-07 17:46:54 DEBUG log:70 - Starting org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298
2015-03-07 17:46:54 DEBUG log:70 - Starting org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192
2015-03-07 17:46:54 DEBUG log:70 - Starting org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137
2015-03-07 17:46:54 DEBUG log:70 - STARTED ServletContextHandler@593114bf@593114bf/,null
2015-03-07 17:46:54 DEBUG log:70 - Starting ResourceHandler@25493558
2015-03-07 17:46:54 DEBUG log:70 - starting ResourceHandler@25493558
2015-03-07 17:46:54 DEBUG log:70 - STARTED ResourceHandler@25493558
2015-03-07 17:46:54 DEBUG log:70 - starting HandlerList@50dafd61
2015-03-07 17:46:54 DEBUG log:70 - STARTED HandlerList@50dafd61
2015-03-07 17:46:54 DEBUG log:70 - starting org.eclipse.jetty.server.Server@16f376a3
2015-03-07 17:46:54 DEBUG log:70 - Starting SelectChannelConnector@0.0.0.0:8081
2015-03-07 17:46:54 DEBUG log:70 - Starting org.eclipse.jetty.server.nio.SelectChannelConnector$1@4b70ca96
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.eclipse.jetty.server.nio.SelectChannelConnector$1@4b70ca96
2015-03-07 17:46:54 INFO log:55 - Started SelectChannelConnector@0.0.0.0:8081
2015-03-07 17:46:54 DEBUG log:70 - STARTED SelectChannelConnector@0.0.0.0:8081
2015-03-07 17:46:54 DEBUG log:70 - STARTED org.eclipse.jetty.server.Server@16f376a3
2015-03-07 17:46:54 INFO WebInfoServer:190 - Started web info server for JobManager on null:8081
org.eclipse.jetty.server.Server@16f376a3 STOPPED
+-HandlerList@50dafd61 started
+-ServletContextHandler@593114bf@593114bf/,null started
| +-SessionHandler@679a7ae9 started
| +-ServletHandler@4efb8c22 started
| +-[/jobsInfo]=>org.apache.flink.runtime.jobmanager.web.JobManagerInfoServlet-116914007{}
| +-[/logInfo]=>org.apache.flink.runtime.jobmanager.web.LogfileInfoServlet-243118298{}
| +-[/setupInfo]=>org.apache.flink.runtime.jobmanager.web.SetupInfoServlet-1420072137{}
| +-[/menu]=>org.apache.flink.runtime.jobmanager.web.MenuServlet-994616192{}
+-ResourceHandler@25493558 started