ResultPartitionMetrics

[aitozi]

Hi,

i see in version 1.3, it add the ResultPartitionMetrics with
issue:https://issues.apache.org/jira/browse/FLINK-5090

but i am doubt what is the difference between totalQueueLen and
inputQueueLength in
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network

i read the code where register the both metric in Task.java:

this.metrics.getIOMetricGroup().initializeBufferMetrics(this);  

// register detailed network metrics, if configured
if
(taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
{
                                // similar to MetricUtils.instantiateNetworkMetrics() but inside this
IOMetricGroup
                                MetricGroup networkGroup =
this.metrics.getIOMetricGroup().addGroup("Network");
                                MetricGroup outputGroup = networkGroup.addGroup("Output");
                                MetricGroup inputGroup = networkGroup.addGroup("Input");

                                // output metrics
                                for (int i = 0; i < producedPartitions.length; i++) {
                                        ResultPartitionMetrics.registerQueueLengthMetrics(
                                                outputGroup.addGroup(i), producedPartitions[i]);
                                }

                                for (int i = 0; i < inputGates.length; i++) {
                                        InputGateMetrics.registerQueueLengthMetrics(
                                                inputGroup.addGroup(i), inputGates[i]);
                                }
                        }

i think the first :initializeBufferMetrics#InputBuffersGauge will get all
the buffers in AllInputGate of the Task and i think the method in
InputGateMetric
"group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
same thing , if i understand wrong , please tell me,.

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Timo Walther]

Hi Aitozi,

I will loop in people that are more familar with the network stack and
metrics. Maybe this is a bug?

Regards,
Timo



Am 10/22/17 um 4:36 PM schrieb aitozi:

> Hi,
>
> i see in version 1.3, it add the ResultPartitionMetrics with
> issue:https://issues.apache.org/jira/browse/FLINK-5090
>
> but i am doubt what is the difference between totalQueueLen and
> inputQueueLength in
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network
>
> i read the code where register the both metric in Task.java:
>
> this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
>
> // register detailed network metrics, if configured
> if
> (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
> {
> // similar to MetricUtils.instantiateNetworkMetrics() but inside this
> IOMetricGroup
> MetricGroup networkGroup =
> this.metrics.getIOMetricGroup().addGroup("Network");
> MetricGroup outputGroup = networkGroup.addGroup("Output");
> MetricGroup inputGroup = networkGroup.addGroup("Input");
>
> // output metrics
> for (int i = 0; i < producedPartitions.length; i++) {
> ResultPartitionMetrics.registerQueueLengthMetrics(
> outputGroup.addGroup(i), producedPartitions[i]);
> }
>
> for (int i = 0; i < inputGates.length; i++) {
> InputGateMetrics.registerQueueLengthMetrics(
> inputGroup.addGroup(i), inputGates[i]);
> }
> }
>
> i think the first :initializeBufferMetrics#InputBuffersGauge will get all
> the buffers in AllInputGate of the Task and i think the method in
> InputGateMetric
> "group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
> same thing , if i understand wrong , please tell me,.
>
> thanks,
> Aitozi
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

The metrics registered in initializeBufferMetrics aggregate across all InputGates, whereas the metrics registered in the InputGateMetrics are separate for each InputGate.

As an example, let's say a task has 2 input gates, with each having 2 input buffers queued:

// IOMetricGroup#initializeBufferMetrics
task.buffers.inputQueueLength = 4

// detailed metrics
task.input.0.totalQueueLen = 2
task.input.1.totalQueueLen = 2

On 23.10.2017 09:31, Timo Walther wrote:

Hi Aitozi,

I will loop in people that are more familar with the network stack and metrics. Maybe this is a bug?

Regards,
Timo



Am 10/22/17 um 4:36 PM schrieb aitozi:

Hi,

i see in version 1.3, it add the ResultPartitionMetrics with
issue:https://issues.apache.org/jira/browse/FLINK-5090

but i am doubt what is the difference between totalQueueLen and
inputQueueLength in
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network

i read the code where register the both metric in Task.java:

this.metrics.getIOMetricGroup().initializeBufferMetrics(this);

// register detailed network metrics, if configured
if
(taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
{
                // similar to MetricUtils.instantiateNetworkMetrics() but inside this
IOMetricGroup
                MetricGroup networkGroup =
this.metrics.getIOMetricGroup().addGroup("Network");
                MetricGroup outputGroup = networkGroup.addGroup("Output");
                MetricGroup inputGroup = networkGroup.addGroup("Input");

                // output metrics
                for (int i = 0; i < producedPartitions.length; i++) {
                    ResultPartitionMetrics.registerQueueLengthMetrics(
                        outputGroup.addGroup(i), producedPartitions[i]);
                }

                for (int i = 0; i < inputGates.length; i++) {
                    InputGateMetrics.registerQueueLengthMetrics(
                        inputGroup.addGroup(i), inputGates[i]);
                }
            }

i think the first :initializeBufferMetrics#InputBuffersGauge will get all
the buffers in AllInputGate of the Task and i think the method in
InputGateMetric
"group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
same thing , if i understand wrong , please tell me,.

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[aitozi]

hi,

i have understood it.

thanks,
aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Nico Kruber]

Hi Aitozi,
the difference is the scope: the normal metrics (without
taskmanager.net.detailed-metrics) reflect _all_ buffers of a task while the
detailed statistics are more fine-grained and give you statistics per input (or
output) gate - the "total" there reflects the fact that each gate has multiple
channels and the metrics offered are the sum among all of them.


Nico

On Monday, 23 October 2017 09:31:04 CET Timo Walther wrote:

> Hi Aitozi,
>
> I will loop in people that are more familar with the network stack and
> metrics. Maybe this is a bug?
>
> Regards,
> Timo
>
> Am 10/22/17 um 4:36 PM schrieb aitozi:
> > Hi,
> >
> > i see in version 1.3, it add the ResultPartitionMetrics with
> > issue:https://issues.apache.org/jira/browse/FLINK-5090
> >
> > but i am doubt what is the difference between totalQueueLen and
> > inputQueueLength in
> > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/met
> > rics.html#network
> >
> > i read the code where register the both metric in Task.java:
> >
> > this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
> >
> > // register detailed network metrics, if configured
> > if
> > (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWOR
> > K_DETAILED_METRICS)) {
> >
> > // similar to MetricUtils.instantiateNetworkMetrics() but inside

this

> >
> > IOMetricGroup
> >
> > MetricGroup networkGroup =
> >
> > this.metrics.getIOMetricGroup().addGroup("Network");
> >
> > MetricGroup outputGroup = networkGroup.addGroup("Output");
> > MetricGroup inputGroup = networkGroup.addGroup("Input");
> >
> > // output metrics
> > for (int i = 0; i < producedPartitions.length; i++) {
> >
> > ResultPartitionMetrics.registerQueueLengthMetrics(
> >
> > outputGroup.addGroup(i), producedPartitions[i]);
> >
> > }
> >
> > for (int i = 0; i < inputGates.length; i++) {
> >
> > InputGateMetrics.registerQueueLengthMetrics(
> >
> > inputGroup.addGroup(i), inputGates[i]);
> >
> > }
> >
> > }
> >
> > i think the first :initializeBufferMetrics#InputBuffersGauge will get all
> > the buffers in AllInputGate of the Task and i think the method in
> > InputGateMetric
> > "group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
> > same thing , if i understand wrong , please tell me,.
> >
> > thanks,
> > Aitozi
> >
> >
> >
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/