Web Dashboard reports 0 for sent/received bytes and records

[Mohammad Kargar]

Hi Konstantin,

What you are explaining here makes sense from a technical perspective! However, functionality wise, I still think it makes sense for the platform to collect and report, somehow, those metrics per job and task out of the box.

Thanks,

Mohammad

---------------------------------------------------------------------------

Hi Mohammad,

that's normal. The source will never have a value other than zero in the
"Records received" column, only in the "Records sent" column. But since
your whole program is chained to one subtask, the source will never send
anything to another subtask, because there is none.
Same for the sink, it will only count values it receives from its
preceding subtask, but due to chaining, there is no preceding subtask.
Thus there will only be zeros in your dashboard.
You can add your own Accumulators or Metrics and see the number of
records in your operators through that, or you could also disable the
chaining (but chaining is a good thing for performance, so you probably
don't want to do that).

Hope that helps,
best regards

Konstantin

On 30.03.2017 16:48, Mohammad Kargar wrote:
> Hi,
> 
> I have a simple job where FlinkKafkaConsumer08 reads data from a topic
> (with 8 partitions) and a custom Sink implementation inserts the data
> into Accumulo database. I can verify that the data is getting inserted
> into the db by adding simple system.out.println(...) to the invoke
> method of the sink and also monitoring Accumulo stats. However Flink Web
> Dashboard reports 0 for sent/received bytes and records (see attached
> file for a snapshot). Any thoughts?
> 
> *Hadoop platform*: Hortonworks Data Platform 2.5
> *Flink*: flink-1.2.0-bin-hadoop27-scala_2.10
> 
> Thanks,
> Mohammad

-- 
Konstantin Gregor * [hidden email]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

[Chesnay Schepler]

Hello,

there are collected and are accessible under the metrics tab.

Currently we can't display them directly since the backing job data-structure only sees tasks(aka chains) and
not the individual operators. There are some ongoing efforts to change that though, so we may be able
to improve the display in the web UI as well soon!

Regards,
Chesnay

On 31.03.2017 16:46, Mohammad Kargar wrote:

Hi Konstantin,

What you are explaining here makes sense from a technical perspective! However, functionality wise, I still think it makes sense for the platform to collect and report, somehow, those metrics per job and task out of the box.

Thanks,

Mohammad

---------------------------------------------------------------------------

Hi Mohammad,

that's normal. The source will never have a value other than zero in the
"Records received" column, only in the "Records sent" column. But since
your whole program is chained to one subtask, the source will never send
anything to another subtask, because there is none.
Same for the sink, it will only count values it receives from its
preceding subtask, but due to chaining, there is no preceding subtask.
Thus there will only be zeros in your dashboard.
You can add your own Accumulators or Metrics and see the number of
records in your operators through that, or you could also disable the
chaining (but chaining is a good thing for performance, so you probably
don't want to do that).

Hope that helps,
best regards

Konstantin

On 30.03.2017 16:48, Mohammad Kargar wrote:
> Hi,
> 
> I have a simple job where FlinkKafkaConsumer08 reads data from a topic
> (with 8 partitions) and a custom Sink implementation inserts the data
> into Accumulo database. I can verify that the data is getting inserted
> into the db by adding simple system.out.println(...) to the invoke
> method of the sink and also monitoring Accumulo stats. However Flink Web
> Dashboard reports 0 for sent/received bytes and records (see attached
> file for a snapshot). Any thoughts?
> 
> *Hadoop platform*: Hortonworks Data Platform 2.5
> *Flink*: flink-1.2.0-bin-hadoop27-scala_2.10
> 
> Thanks,
> Mohammad

-- 
Konstantin Gregor * [hidden email]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082