time measured for each iteration in KMeans

[Pa Rö]

hello flink community,

i have write a k means app for clustering temporal geo data. now i want know how many time flink need for compute one iteration. Is it possible to measure that, cause of the execution engine of flink?

best regards,

paul

[rmetzger0]

Hi,

The TaskManager which is running the Sync task is logging when its starting the next iteration. I know its not very convenient.

You can also log the time and Iteration id (from the IterationRuntimeContext) in the open() method.

On Fri, Jun 26, 2015 at 9:57 AM, Pa Rö <[hidden email]> wrote:

hello flink community,

i have write a k means app for clustering temporal geo data. now i want know how many time flink need for compute one iteration. Is it possible to measure that, cause of the execution engine of flink?

best regards,

paul

[Pa Rö]

Thank you for your quick response.
I just do not understand quite what you mean. I need to define a method in Java or do you mean that I should use the monitoring in the browser?

2015-06-26 10:09 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

The TaskManager which is running the Sync task is logging when its starting the next iteration. I know its not very convenient.

You can also log the time and Iteration id (from the IterationRuntimeContext) in the open() method.

On Fri, Jun 26, 2015 at 9:57 AM, Pa Rö <[hidden email]> wrote:

hello flink community,

i have write a k means app for clustering temporal geo data. now i want know how many time flink need for compute one iteration. Is it possible to measure that, cause of the execution engine of flink?

best regards,

paul

[Pa Rö]

now i have implement a time logger in the open and close methods, it is wrok fine, but i try to initial the flink class with a parameter (counter of benchmark round),

but it will initial always with 0. but i get no exception. what i do wrong?

my benchmark class:

public class FlinkBenchmarkLauncher {

    private static List<Long> times = new ArrayList<Long>();

    public static void main(String[] args) throws Exception {
        Properties pro = new Properties();
        pro.load(FlinkBenchmarkLauncher.class.getResourceAsStream("/config.properties"));
        int benchRounds = Integer.parseInt(pro.getProperty("benchmark.rounds"));
       
        FileSystem fs = LocalFileSystem.get(new URI(pro.getProperty("hdfs.namenode")),new org.apache.hadoop.conf.Configuration());
        String outputPath = fs.getHomeDirectory()+pro.getProperty("flink.output");
               
        for(int i=0;i<benchRounds;i++) {
            FlinkMain fm = new FlinkMain((i+1));
           
            long start = System.nanoTime();
            fm.run();
            long end = System.nanoTime();
           
            times.add(end-start);
            if(i!=benchRounds-1) {
                fs.delete(new Path(outputPath+"/points"), false);
                fs.delete(new Path(outputPath+"/centers"), false);
            }
        }
        BenchmarkHelper.writeTimekeepingToFile(times, "flink_benchmark");
    }
}

my flink main class.

public class FlinkMain implements Runnable{
   
    private static int benchmarkCounter;
    private static int iterationCounter = 1;
   
    FlinkMain(int benchmarkCounter) {
        this.benchmarkCounter = benchmarkCounter;
    }
       
    public void run() {   

    // my execute
   ...

2015-06-26 12:34 GMT+02:00 Pa Rö <[hidden email]>:

Thank you for your quick response.
I just do not understand quite what you mean. I need to define a method in Java or do you mean that I should use the monitoring in the browser?

2015-06-26 10:09 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

The TaskManager which is running the Sync task is logging when its starting the next iteration. I know its not very convenient.

You can also log the time and Iteration id (from the IterationRuntimeContext) in the open() method.

On Fri, Jun 26, 2015 at 9:57 AM, Pa Rö <[hidden email]> wrote:

hello flink community,

i have write a k means app for clustering temporal geo data. now i want know how many time flink need for compute one iteration. Is it possible to measure that, cause of the execution engine of flink?

best regards,

paul

[Till Rohrmann]

Do you also have the rest of the code. It would be helpful in order to find out why it's not working.

Cheers,

TIll

On Wed, Jul 1, 2015 at 1:31 PM, Pa Rö <[hidden email]> wrote:

now i have implement a time logger in the open and close methods, it is wrok fine, but i try to initial the flink class with a parameter (counter of benchmark round),

but it will initial always with 0. but i get no exception. what i do wrong?

my benchmark class:

public class FlinkBenchmarkLauncher {

    private static List<Long> times = new ArrayList<Long>();

    public static void main(String[] args) throws Exception {
        Properties pro = new Properties();
        pro.load(FlinkBenchmarkLauncher.class.getResourceAsStream("/config.properties"));
        int benchRounds = Integer.parseInt(pro.getProperty("benchmark.rounds"));
       
        FileSystem fs = LocalFileSystem.get(new URI(pro.getProperty("hdfs.namenode")),new org.apache.hadoop.conf.Configuration());
        String outputPath = fs.getHomeDirectory()+pro.getProperty("flink.output");
               
        for(int i=0;i<benchRounds;i++) {
            FlinkMain fm = new FlinkMain((i+1));
           
            long start = System.nanoTime();
            fm.run();
            long end = System.nanoTime();
           
            times.add(end-start);
            if(i!=benchRounds-1) {
                fs.delete(new Path(outputPath+"/points"), false);
                fs.delete(new Path(outputPath+"/centers"), false);
            }
        }
        BenchmarkHelper.writeTimekeepingToFile(times, "flink_benchmark");
    }
}

my flink main class.

public class FlinkMain implements Runnable{
   
    private static int benchmarkCounter;
    private static int iterationCounter = 1;
   
    FlinkMain(int benchmarkCounter) {
        this.benchmarkCounter = benchmarkCounter;
    }
       
    public void run() {   

    // my execute
   ...

2015-06-26 12:34 GMT+02:00 Pa Rö <[hidden email]>:

Thank you for your quick response.
I just do not understand quite what you mean. I need to define a method in Java or do you mean that I should use the monitoring in the browser?

2015-06-26 10:09 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

The TaskManager which is running the Sync task is logging when its starting the next iteration. I know its not very convenient.

You can also log the time and Iteration id (from the IterationRuntimeContext) in the open() method.

On Fri, Jun 26, 2015 at 9:57 AM, Pa Rö <[hidden email]> wrote:

hello flink community,

i have write a k means app for clustering temporal geo data. now i want know how many time flink need for compute one iteration. Is it possible to measure that, cause of the execution engine of flink?

best regards,

paul