Flink execution time benchmark.

[Saleh]

Hi everyone,

This is a repost as it didn't receive any reply earlier. If anyone is able to provide any insight, much appreciation for that. I am using Flink Streaming API and try to measure the execution time for a word count job. I have tried a method mentioned in one of the similar posts (using JobExecutionResult object) as shown below, but I can not figure out how to find the results as any code after env.execute() does not get executed.

                JobExecutionResult execResult = env.execute();
                System.out.println("Execution time = " + execResult.getNetRuntime());

I have also tried to use "StreamExecutionEnvironment.execute("flinkJob"). The API states that the program execution will be logged and displayed with the provided name "flinkJob" in this case. My question is where can I locate this log file. I am running the Flink jobs on eclipse with maven plugins using just my local machine.

Cheers.

[rmetzger0]

Hi Saleh,
I'm sorry that nobody replied to your message. I think you were not subscribed to the [e-mail]. mailing list when you posted this question. Therefore, we Flink community didn't receive your message.

Did you resolve the issue in the meantime or are you still seeking for help?

[Saleh]

Hi rmetzger0,

Thanx for the response. I didn't know that I had to register before I could receive responses for my posts.
Now I am registered. But the problem is not resolved yet. I know it might not be intuitive to get execution time from a long running streaming job but it is possible to get total execution time after let say I stop the program from running. Can I programatically compute this information? Or can I retrieve it from Flink web UI?

cheers.

[rmetzger0]

Hi Saleh,

The new web interface in Flink 0.10 has also a REST API that you can use for querying job information.

GET http://localhost:8081/jobs/83547b683ad5b388355a49911168fbc7

will give you the following JSON object:

{

   "jid":"83547b683ad5b388355a49911168fbc7",

   "name":"com.dataartisans.Forever",

   "state":"RUNNING",

   "start-time":1447947758093,

   "end-time":-1,

   "duration":89016,

   "now":1447947847109,

   "timestamps":{

      "CREATED":1447947758093,

      "RUNNING":1447947758106,

      "FAILING":0,

      "FAILED":0,

      "CANCELLING":0,

      "CANCELED":0,

      "FINISHED":0,

      "RESTARTING":0

   },

   "vertices":[

      {

         "id":"6005df847512e1ce9d49f591423a60f0",

         "name":"Source: Custom Source -> Flat Map -> Sink: Unnamed",

         "parallelism":1,

         "status":"RUNNING",

         "start-time":1447947758109,

         "end-time":-1,

         "duration":89000,

         "tasks":{

            "CREATED":0,

            "SCHEDULED":0,

            "DEPLOYING":0,

            "RUNNING":1,

            "FINISHED":0,

            "CANCELING":0,

            "CANCELED":0,

            "FAILED":0

         },

         "metrics":{

            "read-bytes":0,

            "write-bytes":0,

            "read-records":0,

            "write-records":0

         }

      }

   ],

   "status-counts":{

      "CREATED":0,

      "SCHEDULED":0,

      "DEPLOYING":0,

      "RUNNING":1,

      "FINISHED":0,

      "CANCELING":0,

      "CANCELED":0,

      "FAILED":0

   },

   "plan":{

      "jid":"83547b683ad5b388355a49911168fbc7",

      "name":"com.dataartisans.Forever",

      "nodes":[

         {

            "id":"6005df847512e1ce9d49f591423a60f0",

            "parallelism":1,

            "operator":"(not set)",

            "operator_strategy":"(not set)",

            "description":"Source: Custom Source -> Flat Map -> Sink: Unnamed",

            "optimizer_properties":{

            }

         }

      ]

   }

}

With that, you can compute the difference between "start-time" and "now".

Regards,

Robert

On Wed, Nov 18, 2015 at 10:17 PM, Saleh <[hidden email]> wrote:

Hi rmetzger0,

Thanx for the response. I didn't know that I had to register before I could
receive responses for my posts.
Now I am registered. But the problem is not resolved yet. I know it might
not be intuitive to get execution time from a long running streaming job but
it is possible to get total execution time after let say I stop the program
from running. Can I programatically compute this information? Or can I
retrieve it from Flink web UI?

cheers.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-execution-time-benchmark-tp3258p3573.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.