Cannot start FlinkMiniCluster.WebServer using different port in FlinkMiniCluster

[Hung]

Hi,

The original port is used so I'm changing the web port but it fails to. Can I ask which part I made a mistake?

The error:

Exception in thread "main" java.lang.NullPointerException
    at org.apache.flink.runtime.minicluster.FlinkMiniCluster.startWebServer(FlinkMiniCluster.scala:295)
    at org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:268)
    at org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:226)
    at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:101)
    at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1181)

The code that changing the configuration of execution environment.

    Configuration env_config = new Configuration();
    env_config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);

    env_config.setString(ConfigConstants.JOB_MANAGER_WEB_PORT_KEY, "9999");

    env_config.setString(ConfigConstants.JOB_MANAGER_WEB_LOG_PATH_KEY, "./data/dummyLogFile.txt");

    StreamExecutionEnvironment env = new LocalStreamEnvironment(env_config);

   /* Some code here
          ....*/

    env.execute();

Flink version is 0.10.1.

Best,

Sendoh

[Till Rohrmann]

It seems that the web server could not been instantiated. The reason for this problem should be in your logs. Could you look it up and post the reason here?

Additionally, we should build in a sanity check to avoid the NPE.

Cheers,

Till

On Wed, Jan 20, 2016 at 5:06 PM, HungChang <[hidden email]> wrote:

The original port is used so I'm changing the web port but it fails to. Can I
ask which part I made a mistake?

The error:

Exception in thread "main" java.lang.NullPointerException
    at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.startWebServer(FlinkMiniCluster.scala:295)
    at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:268)
    at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:226)
    at
org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:101)
    at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1181)

The code that changing the configuration of execution environment.

    Configuration env_config = new Configuration();
    env_config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);

    env_config.setString(ConfigConstants.JOB_MANAGER_WEB_PORT_KEY, "9999");

    env_config.setString(ConfigConstants.JOB_MANAGER_WEB_LOG_PATH_KEY,
"./data/dummyLogFile.txt");

    StreamExecutionEnvironment env = new LocalStreamEnvironment(env_config);

   /* Some code here
          ....*/

    env.execute();

Flink version is 0.10.1.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Hung]

Yea I'm wondering why the web server cannot be instantiated because changing the port 8081 to 9999 works well in the following demo sample of Flink.
https://github.com/dataArtisans/flink-streaming-demo/blob/master/src/main/scala/com/dataartisans/flink_demo/utils/DemoStreamEnvironment.scala

so is the code of Flink syntax correct and I should try to find other issues?

Best,

Sendoh

[Till Rohrmann]

I guess it’s easiest to simply enable logging and see what the problem is. If you run it from the IDE then you can also set a breakpoint in WebMonitorUtils.startWebRuntimeMonitor and see what the exception is.

Cheers,
Till

​

On Wed, Jan 20, 2016 at 6:04 PM, HungChang <[hidden email]> wrote:

Yea I'm wondering why the web server cannot be instantiated because changing
the port 8081 to 9999 works well in the following demo sample of Flink.
https://github.com/dataArtisans/flink-streaming-demo/blob/master/src/main/scala/com/dataartisans/flink_demo/utils/DemoStreamEnvironment.scala

so is the code of Flink syntax correct and I should try to find other
issues?

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414p4420.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Stephan Ewen]

I think this null pointer comes when the log files are not found (bug in 0.10).

You can double check by either trying 1.0-SNAPSHOT or putting for test an absolute path of a file that exists for the log file.

Greetings,

Stephan

On Wed, Jan 20, 2016 at 6:33 PM, Till Rohrmann <[hidden email]> wrote:

I guess it’s easiest to simply enable logging and see what the problem is. If you run it from the IDE then you can also set a breakpoint in WebMonitorUtils.startWebRuntimeMonitor and see what the exception is.

Cheers,
Till

​

On Wed, Jan 20, 2016 at 6:04 PM, HungChang <[hidden email]> wrote:

Yea I'm wondering why the web server cannot be instantiated because changing
the port 8081 to 9999 works well in the following demo sample of Flink.
https://github.com/dataArtisans/flink-streaming-demo/blob/master/src/main/scala/com/dataartisans/flink_demo/utils/DemoStreamEnvironment.scala

so is the code of Flink syntax correct and I should try to find other
issues?

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414p4420.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Hung]

Thanks for your suggestion. I have some questions to start WebRuntimeMonitor.

In startWebRuntimeMonitor what should be called for
    - leaderRetrievalService: LeaderRetrievalService,
    -  actorSystem: ActorSystem ?

My ref: (https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.html) Is there any example code using WebRuntimeMonitor?

The file existence check for the dashboard log passes as the following code

// File existence check
        File f = new File(logPath);
        if(f.exists() && !f.isDirectory()) {
            System.out.println("File exists in " + f.getAbsolutePath());
        }

env_config.setString(ConfigConstants.JOB_MANAGER_WEB_LOG_PATH_KEY, logPath);


Best,

Sendoh

[Till Rohrmann]

leaderRetrievalService will retrieve the leading JobManager. Take a look at LeaderRetrievalUtils in order to see how it is created and what options are supported. actorSystem is the ActorSystem which is used to resolve the leader’s Akka URL into an ActorRef. You can simply create one or use an existing one if it is at hand.

I’m wondering why you want to create a WebMonitor yourself?

Cheers,
Till

​

On Thu, Jan 21, 2016 at 12:55 PM, HungChang <[hidden email]> wrote:

Thanks for your suggestion. I have some questions to start WebRuntimeMonitor.

In startWebRuntimeMonitor what should be called for
    - leaderRetrievalService: LeaderRetrievalService,
    -  actorSystem: ActorSystem ?

My ref:
(https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.html)
Is there any example code using WebRuntimeMonitor?

The file check for the dashboard log passed.

Best,

Sendoh




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414p4445.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Hung]

Thanks for your reply.

Yea I'm not sure how to use WebMonitor. For me it's about to write the log into a file in disk that should go to the job manager originally at localhost:8081.

Could you please give a brief example how to use it?

Best,

Sendoh

[Hung]

The following message is obtained after putting BasicConfigurator.configure() in main();
But I don't understand the reason `flink-runtime-web is not in the classpath`.

For me the strange part is using the scala version works well whereas my java version throws exception.

1413 [main] ERROR org.apache.flink.runtime.webmonitor.WebMonitorUtils  - Could not load web runtime monitor. Probably reason: flink-runtime-web is not in the classpath
1414 [main] DEBUG org.apache.flink.runtime.webmonitor.WebMonitorUtils  - Caught exception
java.lang.ClassNotFoundException: org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:130)
        at org.apache.flink.runtime.minicluster.FlinkMiniCluster.startWebServer(FlinkMiniCluster.scala:292)
        at org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:268)
        at org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:226)
        at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:101)

Best,

Sendoh

[Till Rohrmann]

Could you add flink-runtime-web to your dependencies of your project? It seems as if it is missing in your project.

Cheers,
Till

​

On Thu, Jan 21, 2016 at 4:45 PM, HungChang <[hidden email]> wrote:

The following message is obtained after putting BasicConfigurator.configure()
in main();
But I don't understand the reason `flink-runtime-web is not in the
classpath`.

For me the strange part is using the scala version works well whereas my
java version throws exception.

1413 [main] ERROR org.apache.flink.runtime.webmonitor.WebMonitorUtils  -
Could not load web runtime monitor. Probably reason: flink-runtime-web is
not in the classpath
1414 [main] DEBUG org.apache.flink.runtime.webmonitor.WebMonitorUtils  -
Caught exception
java.lang.ClassNotFoundException:
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at
org.apache.flink.runtime.webmonitor.WebMonitorUtils.startWebRuntimeMonitor(WebMonitorUtils.java:130)
        at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.startWebServer(FlinkMiniCluster.scala:292)
        at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:268)
        at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.start(FlinkMiniCluster.scala:226)
        at
org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:101)

Best,

Sendoh




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414p4453.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Hung]

After adding the dependency it totally works! Thank you a lot!

[Till Rohrmann]

Great to hear :-) 

On Thu, Jan 21, 2016 at 4:55 PM, HungChang <[hidden email]> wrote:

After adding the dependency it totally works! Thank you a lot!



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-start-FlinkMiniCluster-WebServer-using-different-port-in-FlinkMiniCluster-tp4414p4455.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.