record processing counter

[Stefano Bortoli]

Hi all,

working with the dashboard I found extremely nice to have a monitor of the tasks processing. I wonder whether it is feasible, or could be interesting, to have a counter associated with the processing of each of the functions. I would not mind to have a little overhead in the overall process if I could understand where I am and be able to estimate the process duration. :-)

what do you think?

saluti,
Stefano

[Ufuk Celebi]

Hey Stefano,

this is definitely possible and also not hard to count. See the RecordWriter class, which emits all records to the network runtime. The problem at the moment is that we don't have a central service or the like, which would be able to aggregate these kinds of runtime information and then (for example) have them displayed on the web interface.

This is on the agenda and in my opinion is very important.

If this is important for you, I could add debug log messages or so, which you would then have to parse on your own. Would that be feasible as a short term solution?

On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

working with the dashboard I found extremely nice to have a monitor of the tasks processing. I wonder whether it is feasible, or could be interesting, to have a counter associated with the processing of each of the functions. I would not mind to have a little overhead in the overall process if I could understand where I am and be able to estimate the process duration. :-)

what do you think?

saluti,
Stefano

[Stefano Bortoli]

Hi Ufuk,

thanks for the tip. I am already happy to know it will be on dashboard one day. Shall I open a ticket/feature request?

For the moment I rely on old school note of execution time. The dataset will not change much, and I can have a reference once I get it to complete the first overall process. :-)

saluti,
Stefano

2014-11-13 15:46 GMT+01:00 Ufuk Celebi <[hidden email]>:

Hey Stefano,

this is definitely possible and also not hard to count. See the RecordWriter class, which emits all records to the network runtime. The problem at the moment is that we don't have a central service or the like, which would be able to aggregate these kinds of runtime information and then (for example) have them displayed on the web interface.

This is on the agenda and in my opinion is very important.

If this is important for you, I could add debug log messages or so, which you would then have to parse on your own. Would that be feasible as a short term solution?

On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

working with the dashboard I found extremely nice to have a monitor of the tasks processing. I wonder whether it is feasible, or could be interesting, to have a counter associated with the processing of each of the functions. I would not mind to have a little overhead in the overall process if I could understand where I am and be able to estimate the process duration. :-)

what do you think?

saluti,
Stefano

[Ufuk Celebi]

Sure, go ahead. :)

I think following issues are related: FLINK-964, FLINK-766. There is also an old GSoC proposal, which is related https://github.com/stratosphere/stratosphere/wiki/GSoC-2014-Project-Proposal-Draft-by-Rajika-Kumarasiri.

It would be cool if you could link the issues and also post the link.

On Thu, Nov 13, 2014 at 4:00 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Ufuk,

thanks for the tip. I am already happy to know it will be on dashboard one day. Shall I open a ticket/feature request?

For the moment I rely on old school note of execution time. The dataset will not change much, and I can have a reference once I get it to complete the first overall process. :-)

saluti,
Stefano

2014-11-13 15:46 GMT+01:00 Ufuk Celebi <[hidden email]>:

Hey Stefano,

this is definitely possible and also not hard to count. See the RecordWriter class, which emits all records to the network runtime. The problem at the moment is that we don't have a central service or the like, which would be able to aggregate these kinds of runtime information and then (for example) have them displayed on the web interface.

This is on the agenda and in my opinion is very important.

If this is important for you, I could add debug log messages or so, which you would then have to parse on your own. Would that be feasible as a short term solution?

On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

working with the dashboard I found extremely nice to have a monitor of the tasks processing. I wonder whether it is feasible, or could be interesting, to have a counter associated with the processing of each of the functions. I would not mind to have a little overhead in the overall process if I could understand where I am and be able to estimate the process duration. :-)

what do you think?

saluti,
Stefano