(DEPRECATED) Apache Flink User Mailing List archive.

Tame Flink UI?

Classic

List

Threaded

5 messages Options

Cliff Resnick

Tame Flink UI?

We're on 1.2-SNAPSHOT, and some time over the past couple of weeks the UI seems to have become much more aggressive polling for metrics. I'm seeing hundreds of 413 errors as the UI continuously tries to GET with a URL over 100k, pretty much overwhelming the SOCKS proxy connection.

Below is an example of a javascript GET that seems to be emitted every few seconds. Is this intentional?

Ufuk Celebi

Re: Tame Flink UI?

Hey Cliff,

yes this has been recently merged to the master branch. I think you are right that this is not feasible. I thought that the metrics are pulled in selectively when you select them via the metrics list. It seems to be not the case.

If it is really the case that everything is always requested then we would have to revert this for the time being. Did you select any metrics manually?

– Ufuk

On 16 November 2016 at 18:40:52, Cliff Resnick ([hidden email]) wrote:

Chesnay Schepler

Re: Tame Flink UI?

Hello,

The WebInterfaces first pulls a list of all available metrics for a
specific taskmanager/job/task (which is reasonable since how else would
you select them),
and then requests the values for all metrics by supplying the name of
every single metric it just received, which is where things get funky.

In this case we have a task with parallelism of around 90 (the number
before the metric name is the subtask index).
Now let's only consider IO metrics (numRecordsIn etc.).
We then have 90 * 6 (task IO metrics) + 90 * 4 (operator IO metrics) * X
(# of operators in the task) metrics.
In the best of case of a single operator this results in 900 metrics
being pulled at once,
which is done every few seconds; i don't know the exact update interval.

We can disable this temporarily in a few ways; the easiest one being to
simply never return any metrics in the initial metrics look-up.
See AbstractMetricsHandler#getAvailableMetricsList

Regards,
Chesnay

On 16.11.2016 19:15, Ufuk Celebi wrote:

Cliff Resnick

Re: Tame Flink UI?

Ufuk,

The above occurs for me simply by selecting a running job from the job list.

Chesnay,

The 413 error is because of the large request size. Given all the repeated parameter names maybe a more compact representation would work? For example, instead of enumerating all metrics, maybe ask for the range?

On Wed, Nov 16, 2016 at 2:05 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

The WebInterfaces first pulls a list of all available metrics for a specific taskmanager/job/task (which is reasonable since how else would you select them),
and then requests the values for all metrics by supplying the name of every single metric it just received, which is where things get funky.

In this case we have a task with parallelism of around 90 (the number before the metric name is the subtask index).
Now let's only consider IO metrics (numRecordsIn etc.).
We then have 90 * 6 (task IO metrics) + 90 * 4 (operator IO metrics) * X (# of operators in the task) metrics.
In the best of case of a single operator this results in 900 metrics being pulled at once,
which is done every few seconds; i don't know the exact update interval.

We can disable this temporarily in a few ways; the easiest one being to simply never return any metrics in the initial metrics look-up.
See AbstractMetricsHandler#getAvailableMetricsList

Regards,
Chesnay

On 16.11.2016 19:15, Ufuk Celebi wrote:

Hey Cliff,

yes this has been recently merged to the master branch. I think you are right that this is not feasible. I thought that the metrics are pulled in selectively when you select them via the metrics list. It seems to be not the case.

If it is really the case that everything is always requested then we would have to revert this for the time being. Did you select any metrics manually?

– Ufuk

On 16 November 2016 at 18:40:52, Cliff Resnick ([hidden email]) wrote:

We're on 1.2-SNAPSHOT, and some time over the past couple of weeks the UI
seems to have become much more aggressive polling for metrics. I'm seeing
hundreds of 413 errors as the UI continuously tries to GET with a URL over
100k, pretty much overwhelming the SOCKS proxy connection.
Below is an example of a javascript GET that seems to be emitted every few
seconds. Is this intentional?
http://ip-10-100-4-12.ec2.internal:20888/proxy/application_1479315304318_0001/jobs/0209c14bd32e15b299330968baedb656/vertices/aa32ec03c20b187e8e543b307f1fed08/metrics?get=0.numBytesInRemotePerSecond,81.numBytesInLocalPerSecond,48.numBytesOutPerSecond,52.buffers.outPoolUsage,136.Sink__bad_records.latency,15.buffers.outPoolUsage,44.Sink__bad_records.numRecordsInPerSecond,38.lastCheckpointSize,76.numBytesInLocal,12.numBytesInRemotePerSecond,97.Sink__bad_records.numRecordsOutPerSecond,126.Sink__bad_records.numRecordsOut,108.numBytesOutPerSecond,138.currentLowWatermark,89.Sink__bad_records.numRecordsOut,61.numBytesInLocal,79.buffers.outputQueueLength,112.buffers.outPoolUsage,50.buffers.outputQueueLength,92.Sink__bad_records.numRecordsIn,0.numBytesInRemote,85.lastCheckpointSize,97.Sink__bad_records.numRecordsInPerSecond,10.numBytesOut,104.numRecordsInPerSecond,64.numBytesInRemotePerSecond,75.Sink__bad_records.numRecordsOut,58.numRecordsOut,64.lastCheckpointSize,143.numBytesInRemote,103.buffers.inputQueueLength,8.buffers.inputQueueLength,117.numRecordsInPerSecond,85.numBytesOutPerSecond,60.numBytesInLocalPerSecond,14.numRecordsIn,83.Sink__bad_records.numRecordsOutPerSecond,8.checkpointAlignmentTime,36.checkpointAlignmentTime,124.buffers.inputQueueLength,26.numRecordsIn,82.numBytesInRemote,104.numRecordsOutPerSecond,77.numBytesInRemote,71.numBytesOutPerSecond,136.Sink__bad_records.numRecordsIn,131.numBytesInLocal,29.buffers.outPoolUsage,1.numBytesInLocalPerSecond,34.numBytesOutPerSecond,106.numRecordsOut,83.buffers.inPoolUsage,4.numBytesOutPerSecond,5.Sink__bad_records.numRecordsOut,34.numRecordsIn,38.numRecordsIn,29.numBytesOut,10.Sink__bad_records.numRecordsOut,138.numBytesInRemote,56.numRecordsOut,131.numBytesOutPerSecond,93.buffers.outputQueueLength,122.buffers.outputQueueLength,128.numBytesInLocal,113.buffers.outputQueueLength,25.numBytesOut,126.buffers.outPoolUsage,48.numRecordsOutPerSecond,141.buffers.inPoolUsage,37.buffers.inPoolUsage,61.Sink__bad_records.numRecordsInPerSecond,14.numBytesOut,67.numRecordsOutPerSecond,90.lastCheckpointSize,100.numBytesOutPerSecond,25.buffers.inputQueueLength,21.numBytesOut,129.buffers.inPoolUsage,136.numBytesInRemotePerSecond,12.Sink__bad_records.latency,40.numRecordsOut,58.numRecordsIn,95.numRecordsOut,76.Sink__bad_records.numRecordsOutPerSecond,97.buffers.outPoolUsage,127.numBytesInRemotePerSecond,27.Sink__bad_records.numRecordsInPerSecond,141.numBytesInRemotePerSecond,18.numBytesOut,66.numRecordsInPerSecond,44.checkpointAlignmentTime,108.numRecordsOut,109.numBytesInRemote,21.Sink__bad_records.latency,18.buffers.inPoolUsage,61.Sink__bad_records.numRecordsOut,109.numBytesInLocalPerSecond,30.numRecordsOutPerSecond,25.numBytesInRemote,142.Sink__bad_records.numRecordsIn,51.numRecordsInPerSecond,30.numBytesInRemote,97.numRecordsOut,62.numRecordsOutPerSecond,30.Sink__bad_records.latency,52.checkpointAlignmentTime,103.Sink__bad_records.numRecordsIn,40.numRecordsInPerSecond,18.numRecordsInPerSecond,20.numBytesOutPerSecond,108.buffers.outputQueueLength,69.Sink__bad_records.numRecordsOutPerSecond,19.currentLowWatermark,76.numBytesInLocalPerSecond,16.numRecordsInPerSecond,135.numBytesInLocalPerSecond,80.checkpointAlignmentTime,62.Sink__bad_records.numRecordsOut,64.numRecordsInPerSecond,70.buffers.inputQueueLength,79.numRecordsInPerSecond,18.numRecordsIn,104.buffers.outputQueueLength,109.numRecordsIn,10.Sink__bad_records.numRecordsIn,54.numBytesInRemote,87.checkpointAlignmentTime,97.numBytesInLocalPerSecond,37.Sink__bad_records.numRecordsIn,96.Sink__bad_records.numRecordsIn,10.numRecordsIn,24.Sink__bad_records.numRecordsOut,17.numRecordsOut,99.numRecordsOutPerSecond,30.buffers.inputQueueLength,105.checkpointAlignmentTime,59.lastCheckpointSize,46.numBytesInRemotePerSecond,22.numRecordsIn,46.buffers.inputQueueLength,117.numRecordsOutPerSecond,91.numBytesOutPerSecond,106.lastCheckpointSize,102.Sink__bad_records.latency,112.numBytesInLocalPerSecond,19.numRecordsOut,42.numRecordsIn,113.Sink__bad_records.numRecordsOut,36.buffers.outputQueueLength,17.numBytesOutPerSecond,0.buffers.inPoolUsage,115.numRecordsInPerSecond,115.numBytesInRemote,82.Sink__bad_records.latency,143.buffers.outPoolUsage,48.numBytesInRemote,81.numRecordsOutPerSecond,1.currentLowWatermark,99.checkpointAlignmentTime,48.Sink__bad_records.numRecordsOutPerSecond,111.lastCheckpointSize,57.numBytesInLocalPerSecond,30.numBytesInRemotePerSecond,136.numRecordsOutPerSecond,141.numRecordsOut,119.Sink__bad_records.numRecordsInPerSecond,38.Sink__bad_records.numRecordsOut,17.lastCheckpointSize,90.Sink__bad_records.numRecordsOutPerSecond,15.lastCheckpointSize,53.Sink__bad_records.numRecordsIn,143.numRecordsOut,18.buffers.outputQueueLength,28.checkpointAlignmentTime,18.Sink__bad_records.numRecordsOutPerSecond,22.lastCheckpointSize,32.buffers.outputQueueLength,64.Sink__bad_records.latency,13.numBytesInLocalPerSecond,137.numBytesInRemote,20.lastCheckpointSize,89.numBytesInRemotePerSecond,45.buffers.inPoolUsage,44.buffers.inputQueueLength,102.numRecordsOut,86.numRecordsIn,60.Sink__bad_records.numRecordsInPerSecond,14.Sink__bad_records.numRecordsIn,60.numBytesInRemotePerSecond,98.Sink__bad_records.latency,83.numBytesInRemote,23.buffers.inputQueueLength,6.buffers.inputQueueLength,46.Sink__bad_records.latency,52.numRecordsOut,68.Sink__bad_records.latency,128.numBytesOutPerSecond,5.buffers.outPoolUsage,99.numRecordsOut,87.numBytesOut,41.numBytesInLocalPerSecond,109.buffers.outPoolUsage,132.buffers.outPoolUsage,56.checkpointAlignmentTime,91.Sink__bad_records.latency,62.lastCheckpointSize,1.numBytesInRemote,72.numBytesOut,105.numRecordsIn,132.Sink__bad_records.numRecordsOutPerSecond,103.buffers.inPoolUsage,87.lastCheckpointSize,21.checkpointAlignmentTime,90.numBytesOut,22.numRecordsOutPerSecond,41.Sink__bad_records.numRecordsOutPerSecond,102.numBytesInRemotePerSecond,129.numRecordsIn,133.checkpointAlignmentTime,119.numRecordsInPerSecond,44.numRecordsOut,142.numRecordsOutPerSecond,127.Sink__bad_records.numRecordsOut,54.buffers.outputQueueLength,65.buffers.inputQueueLength,25.Sink__bad_records.numRecordsOutPerSecond,61.numBytesOut,42.numRecordsOut,95.checkpointAlignmentTime,109.numRecordsOutPerSecond,133.numBytesInLocalPerSecond,16.Sink__bad_records.latency,23.currentLowWatermark,0.Sink__bad_records.numRecordsOutPerSecond,123.numBytesInRemotePerSecond,39.Sink__bad_records.numRecordsOutPerSecond,8.numBytesInLocal,76.Sink__bad_records.numRecordsOut,80.Sink__bad_records.numRecordsInPerSecond,125.numRecordsIn,62.numBytesInLocalPerSecond,5.numRecordsOutPerSecond,11.Sink__bad_records.numRecordsOut,136.numBytesOut,25.Sink__bad_records.latency,58.numBytesOut,69.numBytesOut,30.numRecordsIn,78.numBytesInLocalPerSecond,98.numBytesOut,117.checkpointAlignmentTime,81.numBytesInRemote,83.buffers.outPoolUsage,128.numRecordsInPerSecond,4.lastCheckpointSize,4.Sink__bad_records.numRecordsOut,84.buffers.outputQueueLength,121.checkpointAlignmentTime,60.numBytesOutPerSecond,50.numRecordsIn,27.numRecordsInPerSecond,132.Sink__bad_records.numRecordsIn,81.numRecordsOut,20.currentLowWatermark,62.Sink__bad_records.numRecordsInPerSecond,59.numBytesInRemotePerSecond,126.buffers.outputQueueLength,1.checkpointAlignmentTime,43.numRecordsOutPerSecond,78.Sink__bad_records.numRecordsInPerSecond,64.checkpointAlignmentTime,75.buffers.outputQueueLength,32.Sink__bad_records.numRecordsOutPerSecond,54.numRecordsOut,7.buffers.inPoolUsage,117.buffers.outputQueueLength,7.Sink__bad_records.numRecordsOutPerSecond,141.checkpointAlignmentTime,91.numBytesInRemotePerSecond,13.checkpointAlignmentTime,68.numBytesInRemotePerSecond,132.numBytesInRemotePerSecond,139.Sink__bad_records.numRecordsOutPerSecond,10.Sink__bad_records.numRecordsInPerSecond,100.Sink__bad_records.numRecordsInPerSecond,114.numBytesInRemote,84.currentLowWatermark,122.buffers.inPoolUsage,11.buffers.outputQueueLength,75.numRecordsInPerSecond,22.buffers.inPoolUsage,114.numBytesInLocalPerSecond,59.Sink__bad_records.latency,70.numRecordsIn,139.numBytesInRemote,96.numBytesOutPerSecond,63.Sink__bad_records.numRecordsOut,88.buffers.outputQueueLength,24.numBytesInRemote,63.buffers.outPoolUsage,125.numBytesOut,114.numBytesOut,43.lastCheckpointSize,46.numRecordsOut,47.numBytesOut,39.buffers.inputQueueLength,63.Sink__bad_records.numRecordsIn,36.numBytesOut,51.numBytesInRemotePerSecond,94.numRecordsOutPerSecond,6.buffers.outputQueueLength,21.numBytesInRemotePerSecond,114.numBytesOutPerSecond,79.Sink__bad_records.numRecordsInPerSecond,50.numBytesOut,25.Sink__bad_records.numRecordsOut,89.Sink__bad_records.latency,123.buffers.outPoolUsage,104.numRecordsOut,142.currentLowWatermark,2.buffers.outputQueueLength,42.currentLowWatermark,49.numBytesInRemote,27.Sink__bad_records.numRecordsIn,27.currentLowWatermark,64.buffers.inPoolUsage,137.buffers.outPoolUsage,130.Sink__bad_records.numRecordsOutPerSecond,69.currentLowWatermark,83.lastCheckpointSize,101.Sink__bad_records.numRecordsInPerSecond,98.numBytesInRemotePerSecond,8.currentLowWatermark,126.buffers.inputQueueLength,127.currentLowWatermark,140.numRecordsIn,93.numRecordsOut,55.numBytesInRemotePerSecond,88.currentLowWatermark,127.Sink__bad_records.latency,100.currentLowWatermark,77.numRecordsInPerSecond,83.numBytesInLocalPerSecond,78.numBytesInRemote,51.Sink__bad_records.latency,113.numBytesInRemote,114.Sink__bad_records.numRecordsOut,45.buffers.outputQueueLength,123.currentLowWatermark,49.buffers.outPoolUsage,99.numBytesInLocalPerSecond,36.lastCheckpointSize,46.numRecordsIn,91.numRecordsOut,20.Sink__bad_records.numRecordsIn,103.numBytesOut,75.numRecordsOutPerSecond,77.buffers.outPoolUsage,41.buffers.inPoolUsage,118.numRecordsOut,46.currentLowWatermark,32.checkpointAlignmentTime,0.numRecordsOutPerSecond,65.currentLowWatermark,102.Sink__bad_records.numRecordsInPerSecond,68.numRecordsInPerSecond,0.numBytesOut,3.numBytesInLocal,21.buffers.outPoolUsage,123.Sink__bad_records.latency,132.lastCheckpointSize,41.buffers.outputQueueLength,55.Sink__bad_records.latency,126.numRecordsInPerSecond,82.numBytesOutPerSecond,54.numBytesOutPerSecond,29.numRecordsInPerSecond,35.buffers.outPoolUsage,142.numBytesInRemote,59.numBytesInLocalPerSecond,66.numRecordsIn,53.numBytesInRemote,125.lastCheckpointSize,68.numBytesOutPerSecond,48.numRecordsOut,129.lastCheckpointSize,100.numRecordsOut,104.currentLowWatermark,27.numRecordsOutPerSecond,26.buffers.inPoolUsage,40.numBytesOutPerSecond,123.numRecordsOutPerSecond,8.numBytesOut,39.Sink__bad_records.numRecordsOut,74.Sink__bad_records.numRecordsOutPerSecond,137.Sink__bad_records.numRecordsOutPerSecond,25.numBytesInRemotePerSecond,60.buffers.inPoolUsage,34.Sink__bad_records.numRecordsOutPerSecond,67.buffers.inPoolUsage,108.numRecordsInPerSecond,4.buffers.inPoolUsage,135.numRecordsOut,32.numBytesInLocalPerSecond,92.checkpointAlignmentTime,34.numBytesInRemotePerSecond,17.numBytesInLocal,85.numRecordsOut,129.buffers.outPoolUsage,11.buffers.inputQueueLength,103.Sink__bad_records.numRecordsInPerSecond,7.numBytesInRemotePerSecond,120.numRecordsIn,49.Sink__bad_records.numRecordsOut,52.numBytesInRemote,110.numBytesInLocalPerSecond,116.Sink__bad_records.numRecordsIn,130.numRecordsInPerSecond,143.buffers.outputQueueLength,142.numBytesOutPerSecond,108.currentLowWatermark,66.numRecordsOut,92.buffers.outPoolUsage,16.currentLowWatermark,32.numBytesInRemote,11.Sink__bad_records.numRecordsInPerSecond,20.numBytesInLocal,74.numBytesInLocalPerSecond,43.Sink__bad_records.latency,94.Sink__bad_records.latency,24.Sink__bad_records.numRecordsIn,53.buffers.inputQueueLength,28.Sink__bad_records.latency,10.buffers.outPoolUsage,86.Sink__bad_records.latency,87.Sink__bad_records.numRecordsOut,81.Sink__bad_records.numRecordsOutPerSecond,1.numBytesInLocal,41.numRecordsOutPerSecond,128.checkpointAlignmentTime,43.numBytesOutPerSecond,121.numBytesInLocalPerSecond,121.numRecordsIn,26.Sink__bad_records.numRecordsOut,77.numBytesInRemotePerSecond,106.buffers.inPoolUsage,124.Sink__bad_records.numRecordsOut,117.Sink__bad_records.numRecordsInPerSecond,95.Sink__bad_records.latency,141.buffers.inputQueueLength,100.numBytesOut,71.buffers.outputQueueLength,106.numBytesInRemotePerSecond,58.buffers.outputQueueLength,55.numBytesInRemote,86.numBytesInRemotePerSecond,139.numRecordsInPerSecond,6.numRecordsOut,47.numBytesInRemote,94.numBytesInRemotePerSecond,104.numBytesOut,77.Sink__bad_records.numRecordsOut,4.buffers.inputQueueLength,7.numBytesOutPerSecond,1.numBytesOut,119.numBytesOutPerSecond,125.buffers.inPoolUsage,11.numRecordsOut,8.numRecordsIn,24.checkpointAlignmentTime,18.Sink__bad_records.numRecordsIn,29.numRecordsOut,101.checkpointAlignmentTime,134.buffers.outputQueueLength,86.buffers.inPoolUsage,86.numRecordsInPerSecond,22.numBytesInLocalPerSecond,9.numBytesOutPerSecond,140.numBytesOutPerSecond,100.Sink__bad_records.numRecordsOut,134.Sink__bad_records.numRecordsInPerSecond,88.numBytesOutPerSecond,4.numRecordsIn,92.Sink__bad_records.numRecordsOutPerSecond,55.buffers.outPoolUsage,84.checkpointAlignmentTime,52.Sink__bad_records.latency,62.currentLowWatermark,102.checkpointAlignmentTime,61.buffers.outPoolUsage,117.numBytesOutPerSecond,101.buffers.outPoolUsage,3.Sink__bad_records.numRecordsOut,134.numBytesOutPerSecond,39.numBytesOutPerSecond,41.Sink__bad_records.numRecordsInPerSecond,37.numRecordsOutPerSecond,94.numBytesOutPerSecond,140.buffers.outPoolUsage,93.numRecordsIn,122.numBytesOut,111.numBytesOut,84.numBytesInLocal,82.numRecordsIn,76.numBytesOutPerSecond,99.numBytesInLocal,15.currentLowWatermark,81.numRecordsIn,115.numBytesOut,43.buffers.outPoolUsage,126.numBytesOut,62.buffers.outputQueueLength,3.buffers.outputQueueLength,104.Sink__bad_records.numRecordsOutPerSecond,95.numBytesInRemotePerSecond,34.Sink__bad_records.latency,51.numRecordsOutPerSecond,73.numRecordsInPerSecond,123.Sink__bad_records.numRecordsIn,121.Sink__bad_records.numRecordsInPerSecond,71.Sink__bad_records.numRecordsOutPerSecond,16.checkpointAlignmentTime,64.Sink__bad_records.numRecordsOut,85.numBytesInLocalPerSecond,53.numBytesInLocal,68.numBytesInLocal,113.numRecordsIn,42.buffers.inputQueueLength,71.numBytesInLocal,48.buffers.inPoolUsage,35.numBytesInLocal,131.numRecordsOut,123.Sink__bad_records.numRecordsOutPerSecond,116.numBytesInLocalPerSecond,34.currentLowWatermark,110.numRecordsOutPerSecond,99.Sink__bad_records.numRecordsInPerSecond,72.Sink__bad_records.numRecordsIn,89.numRecordsOut,36.Sink__bad_records.numRecordsOut,128.numBytesInLocalPerSecond,64.Sink__bad_records.numRecordsInPerSecond,100.numRecordsInPerSecond,19.numBytesInRemote,101.currentLowWatermark,25.checkpointAlignmentTime,17.numBytesInLocalPerSecond,129.Sink__bad_records.numRecordsOut,102.lastCheckpointSize,25.numRecordsInPerSecond,57.Sink__bad_records.numRecordsIn,124.Sink__bad_records.latency,29.buffers.inPoolUsage,83.checkpointAlignmentTime,133.numBytesOut,117.buffers.inputQueueLength,6.numRecordsInPerSecond,60.Sink__bad_records.latency,100.buffers.outputQueueLength,132.numRecordsIn,97.numBytesOut,33.Sink__bad_records.numRecordsIn,132.Sink__bad_records.latency,12.buffers.outPoolUsage,45.numBytesOutPerSecond,137.numBytesOut,120.currentLowWatermark,28.numBytesInRemotePerSecond,18.buffers.inputQueueLength,76.Sink__bad_records.numRecordsIn,113.lastCheckpointSize,7.Sink__bad_records.latency,105.numBytesInLocal,81.currentLowWatermark,80.numBytesInRemote,69.numBytesInLocalPerSecond,106.Sink__bad_records.latency,60.Sink__bad_records.numRecordsOutPerSecond,138.numBytesInLocal,20.Sink__bad_records.numRecordsOutPerSecond,123.numBytesInLocal,107.Sink__bad_records.numRecordsIn,88.numRecordsOutPerSecond,92.numRecordsOutPerSecond,141.numBytesInLocal,127.Sink__bad_records.numRecordsIn,73.currentLowWatermark,131.numRecordsOutPerSecond,125.checkpointAlignmentTime,133.buffers.inPoolUsage,17.checkpointAlignmentTime,93.numBytesInLocalPerSecond,83.numBytesOut,57.Sink__bad_records.numRecordsOutPerSecond,125.numBytesOutPerSecond,11.buffers.inPoolUsage,141.numRecordsInPerSecond,108.lastCheckpointSize,109.numBytesInRemotePerSecond,74.numRecordsIn,13.Sink__bad_records.numRecordsOut,121.numBytesInLocal,48.numBytesInLocal,9.currentLowWatermark,111.Sink__bad_records.numRecordsOut,28.numBytesOutPerSecond,74.Sink__bad_records.numRecordsOut,25.numRecordsOut,2.numRecordsOut,79.numBytesOut,15.numRecordsOut,29.numRecordsOutPerSecond,76.numBytesOut,4.numBytesInRemotePerSecond,118.numBytesInLocal,122.buffers.inputQueueLength,47.checkpointAlignmentTime,7.buffers.outPoolUsage,94.numBytesOut,118.Sink__bad_records.latency,134.Sink__bad_records.numRecordsOutPerSecond,85.numRecordsIn,72.buffers.inputQueueLength,86.buffers.inputQueueLength,100.Sink__bad_records.numRecordsOutPerSecond,37.buffers.outPoolUsage,75.buffers.inPoolUsage,90.numBytesInLocalPerSecond,52.numBytesInRemotePerSecond,53.currentLowWatermark,51.numBytesOutPerSecond,136.numBytesInLocal,53.buffers.outputQueueLength,86.numBytesInLocal,43.Sink__bad_records.numRecordsInPerSecond,116.Sink__bad_records.numRecordsOutPerSecond,2.Sink__bad_records.numRecordsOutPerSecond,38.buffers.outPoolUsage,28.Sink__bad_records.numRecordsInPerSecond,110.numBytesInRemotePerSecond,86.buffers.outPoolUsage,3.numBytesInLocalPerSecond,140.numBytesInLocalPerSecond,139.buffers.outputQueueLength,10.buffers.outputQueueLength,117.numRecordsIn,29.buffers.inputQueueLength,66.numBytesInLocal,103.numBytesInLocal,2.Sink__bad_records.numRecordsInPerSecond,135.buffers.outPoolUsage,131.buffers.outputQueueLength,133.buffers.inputQueueLength,67.buffers.outputQueueLength,111.Sink__bad_records.numRecordsOutPerSecond,112.currentLowWatermark,97.lastCheckpointSize,117.numBytesInRemote,139.numRecordsOut,45.Sink__bad_records.numRecordsInPerSecond,111.currentLowWatermark,26.Sink__bad_records.numRecordsInPerSecond,55.checkpointAlignmentTime,65.numBytesOut,54.numBytesOut,54.currentLowWatermark,37.buffers.inputQueueLength,46.Sink__bad_records.numRecordsOutPerSecond,88.buffers.outPoolUsage,131.currentLowWatermark,125.buffers.outputQueueLength,51.numBytesInLocal,55.numRecordsInPerSecond,105.numBytesInLocalPerSecond,115.numBytesInRemotePerSecond,2.lastCheckpointSize,92.currentLowWatermark,124.numRecordsIn,128.numRecordsIn,139.numBytesInLocalPerSecond,109.Sink__bad_records.numRecordsOutPerSecond,124.numBytesInRemotePerSecond,140.buffers.outputQueueLength,6.Sink__bad_records.numRecordsOut,54.numRecordsOutPerSecond,33.numBytesInLocal,121.numRecordsOut,118.numBytesInRemotePerSecond,78.numRecordsIn,76.buffers.outputQueueLength,0.numRecordsIn,36.numBytesInLocalPerSecond,15.numBytesInLocal,56.buffers.inPoolUsage,95.Sink__bad_records.numRecordsInPerSecond,19.numRecordsOutPerSecond,33.buffers.outputQueueLength,85.numBytesInRemote,102.numBytesInLocalPerSecond,37.Sink__bad_records.latency,23.Sink__bad_records.numRecordsOut,116.Sink__bad_records.numRecordsOut,120.numBytesInRemote,75.numBytesInRemote,118.numBytesOut,62.numRecordsInPerSecond,48.buffers.inputQueueLength,129.numBytesOut,46.numBytesOut,60.buffers.outPoolUsage,43.numBytesOut,60.numBytesInRemote,32.numBytesOut,43.numBytesInRemotePerSecond,0.Sink__bad_records.numRecordsInPerSecond,91.checkpointAlignmentTime,79.numBytesOutPerSecond,93.numBytesOutPerSecond,52.lastCheckpointSize,119.currentLowWatermark,101.Sink__bad_records.latency,46.buffers.outPoolUsage,37.numBytesOutPerSecond,135.numBytesInRemote,114.buffers.inPoolUsage,77.Sink__bad_records.latency,7.numRecordsOutPerSecond,115.Sink__bad_records.numRecordsInPerSecond,1.numRecordsInPerSecond,139.Sink__bad_records.numRecordsOut,83.Sink__bad_records.numRecordsIn,96.buffers.outputQueueLength,30.buffers.inPoolUsage,35.currentLowWatermark,112.Sink__bad_records.numRecordsIn,67.buffers.inputQueueLength,44.numRecordsInPerSecond,13.Sink__bad_records.numRecordsOutPerSecond,138.Sink__bad_records.numRecordsInPerSecond,53.Sink__bad_records.numRecordsOutPerSecond,115.Sink__bad_records.latency,21.numRecordsOut,92.numRecordsInPerSecond,27.numBytesInRemote,89.Sink__bad_records.numRecordsIn,111.numRecordsInPerSecond,134.buffers.outPoolUsage,44.Sink__bad_records.numRecordsIn,101.numBytesInRemotePerSecond,141.Sink__bad_records.latency,48.checkpointAlignmentTime,94.buffers.outPoolUsage,4.Sink__bad_records.latency,8.buffers.outPoolUsage,91.buffers.inputQueueLength,60.checkpointAlignmentTime,107.numBytesOut,99.currentLowWatermark,37.numBytesInRemotePerSecond,94.buffers.inPoolUsage,51.Sink__bad_records.numRecordsOut,57.numBytesOut,68.numBytesOut,109.Sink__bad_records.latency,41.lastCheckpointSize,136.Sink__bad_records.numRecordsInPerSecond,6.Sink__bad_records.numRecordsIn,45.numRecordsIn,107.numBytesInRemote,93.Sink__bad_records.numRecordsInPerSecond,55.numBytesInLocalPerSecond,27.buffers.outputQueueLength,141.buffers.outPoolUsage,134.lastCheckpointSize,2.currentLowWatermark,40.Sink__bad_records.numRecordsIn,14.numRecordsInPerSecond,4.numBytesOut,64.numRecordsOutPerSecond,0.Sink__bad_records.numRecordsIn,134.numRecordsInPerSecond,134.numRecordsOutPerSecond,47.Sink__bad_records.numRecordsInPerSecond,16.numRecordsOutPerSecond,123.lastCheckpointSize,64.Sink__bad_records.numRecordsOutPerSecond,24.Sink__bad_records.numRecordsInPerSecond,127.Sink__bad_records.numRecordsOutPerSecond,79.checkpointAlignmentTime,106.numBytesInRemote,56.buffers.inputQueueLength,51.currentLowWatermark,23.buffers.inPoolUsage,76.numRecordsOut,140.numRecordsOutPerSecond,53.lastCheckpointSize,59.numRecordsIn,4.Sink__bad_records.numRecordsIn,62.numBytesInRemote,47.numRecordsIn,135.numBytesInRemotePerSecond,100.lastCheckpointSize,20.numRecordsOut,64.numBytesInLocal,42.buffers.outputQueueLength,87.numBytesOutPerSecond,91.Sink__bad_records.numRecordsIn,52.numRecordsOutPerSecond,22.Sink__bad_records.numRecordsInPerSecond,50.buffers.outPoolUsage,143.lastCheckpointSize,14.Sink__bad_records.numRecordsOut,134.buffers.inputQueueLength,10.checkpointAlignmentTime,13.numRecordsOutPerSecond,125.numRecordsOut,60.buffers.inputQueueLength,92.Sink__bad_records.numRecordsInPerSecond,3.Sink__bad_records.numRecordsInPerSecond,77.buffers.inputQueueLength,143.numBytesOutPerSecond,69.numRecordsIn,121.buffers.inPoolUsage,83.numRecordsInPerSecond,82.buffers.inPoolUsage,9.Sink__bad_records.numRecordsOut,38.numRecordsOut,17.buffers.outPoolUsage,70.numBytesInRemotePerSecond,134.numBytesInLocal,17.buffers.inPoolUsage,57.numRecordsIn,107.buffers.outputQueueLength,95.buffers.outPoolUsage,126.numRecordsOutPerSecond,47.numRecordsInPerSecond,20.Sink__bad_records.latency,90.Sink__bad_records.numRecordsOut,87.buffers.outputQueueLength,91.numBytesInLocalPerSecond,91.numRecordsOutPerSecond,123.Sink__bad_records.numRecordsInPerSecond,140.buffers.inPoolUsage,50.numBytesInLocalPerSecond,79.Sink__bad_records.numRecordsOutPerSecond,137.Sink__bad_records.latency,128.numBytesInRemotePerSecond,124.numRecordsOut,121.numRecordsInPerSecond,125.numBytesInLocal,66.numBytesInLocalPerSecond,86.Sink__bad_records.numRecordsOutPerSecond,107.numRecordsOutPerSecond,141.Sink__bad_records.numRecordsIn,22.numBytesInRemote,27.buffers.outPoolUsage,128.buffers.outPoolUsage,85.Sink__bad_records.numRecordsOut,32.numBytesOutPerSecond,106.checkpointAlignmentTime,85.buffers.outputQueueLength,53.checkpointAlignmentTime,39.numBytesOut,13.buffers.outPoolUsage,37.numRecordsIn,3.numRecordsOutPerSecond,105.Sink__bad_records.numRecordsInPerSecond,82.numRecordsInPerSecond,130.currentLowWatermark,13.numRecordsIn,9.buffers.outputQueueLength,17.numBytesInRemote,13.Sink__bad_records.latency,133.numBytesOutPerSecond,91.currentLowWatermark,38.Sink__bad_records.numRecordsIn,133.numBytesInRemotePerSecond,44.buffers.outputQueueLength,74.Sink__bad_records.latency,100.buffers.inPoolUsage,61.buffers.inputQueueLength,44.buffers.inPoolUsage,11.lastCheckpointSize,99.buffers.outPoolUsage,9.numRecordsOutPerSecond,141.Sink__bad_records.numRecordsOut,139.currentLowWatermark,108.numRecordsIn,136.Sink__bad_records.numRecordsOut,130.buffers.outputQueueLength,58.numBytesInRemotePerSecond,23.buffers.outPoolUsage,31.Sink__bad_records.numRecordsInPerSecond,88.numBytesInLocalPerSecond,72.currentLowWatermark,102.buffers.inPoolUsage,121.numRecordsOutPerSecond,97.numRecordsInPerSecond,97.numBytesOutPerSecond,3.Sink__bad_records.latency,35.numBytesOut,123.numBytesInLocalPerSecond,82.buffers.inputQueueLength,132.checkpointAlignmentTime,48.lastCheckpointSize,44.numBytesOut,78.numRecordsOut,137.numBytesOutPerSecond,79.numRecordsOut,70.currentLowWatermark,89.numRecordsIn,102.Sink__bad_records.numRecordsIn,124.checkpointAlignmentTime,112.buffers.outputQueueLength,58.numRecordsOutPerSecond,44.buffers.outPoolUsage,142.Sink__bad_records.latency,95.Sink__bad_records.numRecordsIn,74.numBytesInRemote,61.checkpointAlignmentTime,79.numRecordsIn,75.lastCheckpointSize,90.numBytesInRemote,54.numBytesInRemotePerSecond,104.numBytesInLocalPerSecond,37.checkpointAlignmentTime,1.numBytesInRemotePerSecond,61.buffers.inPoolUsage,81.numRecordsInPerSecond,28.numBytesOut,66.Sink__bad_records.numRecordsOut,1.buffers.inputQueueLength,34.buffers.inputQueueLength,20.Sink__bad_records.numRecordsOut,77.numRecordsOut,126.Sink__bad_records.latency,45.numBytesInRemote,42.numBytesOutPerSecond,63.buffers.inPoolUsage,71.Sink__bad_records.numRecordsOut,122.numRecordsInPerSecond,11.currentLowWatermark,41.numRecordsIn,19.checkpointAlignmentTime,22.numBytesInRemotePerSecond,129.buffers.inputQueueLength,14.buffers.inputQueueLength,24.numBytesOut,13.numBytesOut,32.currentLowWatermark,48.numRecordsInPerSecond,90.Sink__bad_records.latency,66.numBytesOut,55.numBytesOut,40.numRecordsOutPerSecond,110.Sink__bad_records.latency,86.numBytesInRemote,142.lastCheckpointSize,56.Sink__bad_records.latency,33.lastCheckpointSize,31.lastCheckpointSize,42.buffers.inPoolUsage,30.Sink__bad_records.numRecordsInPerSecond,15.Sink__bad_records.numRecordsOut,40.buffers.outPoolUsage,25.buffers.inPoolUsage,109.buffers.inputQueueLength,118.numBytesInRemote,60.numRecordsOut,17.numBytesOut,26.buffers.outputQueueLength,89.buffers.outPoolUsage,45.numBytesInLocalPerSecond,52.Sink__bad_records.numRecordsIn,72.Sink__bad_records.latency,77.numRecordsOutPerSecond,30.currentLowWatermark,134.numBytesInRemote,13.currentLowWatermark,122.lastCheckpointSize,120.lastCheckpointSize,26.lastCheckpointSize,61.numRecordsOut,106.Sink__bad_records.numRecordsIn,3.buffers.inputQueueLength,71.numRecordsOut,81.checkpointAlignmentTime,34.buffers.outPoolUsage,138.numRecordsOutPerSecond,20.buffers.inputQueueLength,0.checkpointAlignmentTime,56.Sink__bad_records.numRecordsIn,55.buffers.inPoolUsage,45.numBytesInLocal,77.Sink__bad_records.numRecordsIn,122.numBytesInRemote,131.Sink__bad_records.numRecordsIn,76.Sink__bad_records.numRecordsInPerSecond,106.numRecordsOutPerSecond,98.numRecordsInPerSecond,61.Sink__bad_records.latency,0.currentLowWatermark,11.numBytesOutPerSecond,39.Sink__bad_records.numRecordsInPerSecond,18.numRecordsOutPerSecond,89.numBytesInLocalPerSecond,29.Sink__bad_records.numRecordsOut,47.lastCheckpointSize,34.Sink__bad_records.numRecordsOut,12.Sink__bad_records.numRecordsInPerSecond,73.numBytesInRemote,115.numBytesInLocal,97.numRecordsOutPerSecond,13.Sink__bad_records.numRecordsIn,31.buffers.inPoolUsage,72.numRecordsOutPerSecond,112.numRecordsOut,120.numRecordsOutPerSecond,73.numRecordsIn,115.buffers.inputQueueLength,139.numBytesInLocal,60.buffers.outputQueueLength,86.numBytesInLocalPerSecond,61.numRecordsIn,104.Sink__bad_records.numRecordsOut,72.numBytesInLocalPerSecond,130.Sink__bad_records.numRecordsInPerSecond,112.buffers.inputQueueLength,136.buffers.inputQueueLength,76.lastCheckpointSize,51.checkpointAlignmentTime,83.numBytesInLocal,43.currentLowWatermark,113.buffers.inPoolUsage,41.buffers.inputQueueLength,124.Sink__bad_records.numRecordsInPerSecond,81.numBytesInRemotePerSecond,26.numRecordsOut,40.Sink__bad_records.numRecordsInPerSecond,104.Sink__bad_records.numRecordsInPerSecond,106.numBytesInLocal,142.numBytesOut,122.numRecordsOut,25.numBytesOutPerSecond,50.buffers.inPoolUsage,86.Sink__bad_records.numRecordsOut,69.numBytesInLocal,98.lastCheckpointSize,142.numBytesInLocal,91.Sink__bad_records.numRecordsOut,128.buffers.inputQueueLength,5.buffers.outputQueueLength,95.numRecordsIn,104.checkpointAlignmentTime,101.numBytesInLocalPerSecond,24.numRecordsInPerSecond,24.currentLowWatermark,31.Sink__bad_records.latency,39.numBytesInRemote,78.numBytesInLocal,33.checkpointAlignmentTime,99.Sink__bad_records.latency,0.buffers.inputQueueLength,4.numRecordsInPerSecond,69.buffers.outputQueueLength,36.buffers.inPoolUsage,15.Sink__bad_records.numRecordsInPerSecond,72.numBytesInLocal,74.buffers.outputQueueLength,123.buffers.outputQueueLength,117.numBytesInRemotePerSecond,12.numBytesInLocal,20.buffers.outPoolUsage,33.numBytesInRemote,36.numBytesInLocal,98.numBytesInRemote,118.numRecordsIn,78.numRecordsOutPerSecond,55.buffers.outputQueueLength,28.numBytesInLocal,3.buffers.inPoolUsage,15.buffers.inputQueueLength,85.Sink__bad_records.numRecordsOutPerSecond,85.Sink__bad_records.latency,120.numBytesOut,81.buffers.outputQueueLength,114.numRecordsOut,63.checkpointAlignmentTime,65.numBytesInRemotePerSecond,117.numBytesOut,36.buffers.inputQueueLength,128.numBytesOut,28.Sink__bad_records.numRecordsIn,22.numBytesInLocal,98.buffers.outPoolUsage,101.numBytesInLocal,12.buffers.inputQueueLength,34.numBytesInRemote,36.numRecordsOut,31.numBytesInLocal,33.numBytesOut,94.numBytesInLocalPerSecond,31.numRecordsInPerSecond,119.numBytesInRemote,55.buffers.inputQueueLength,76.checkpointAlignmentTime,54.lastCheckpointSize,19.numBytesInLocal,45.Sink__bad_records.latency,135.Sink__bad_records.numRecordsIn,19.Sink__bad_records.latency,129.checkpointAlignmentTime,38.numRecordsInPerSecond,60.numRecordsOutPerSecond,11.numBytesInRemotePerSecond,7.numBytesInLocalPerSecond,57.Sink__bad_records.numRecordsInPerSecond,136.numBytesOutPerSecond,12.numRecordsOutPerSecond,69.lastCheckpointSize,46.numRecordsOutPerSecond,135.numRecordsInPerSecond,103.numBytesInRemotePerSecond,1.numRecordsIn,83.Sink__bad_records.numRecordsInPerSecond,138.buffers.outPoolUsage,32.lastCheckpointSize,97.numBytesInLocal,14.checkpointAlignmentTime,49.Sink__bad_records.latency,98.buffers.outputQueueLength,101.numRecordsOutPerSecond,22.numRecordsInPerSecond,15.numRecordsIn,139.Sink__bad_records.numRecordsIn,30.numRecordsOut,19.buffers.outputQueueLength,134.checkpointAlignmentTime,23.Sink__bad_records.numRecordsIn,91.numBytesInLocal,62.Sink__bad_records.numRecordsIn,55.numBytesInLocal,0.Sink__bad_records.numRecordsOut,139.numBytesOut,29.numBytesInRemotePerSecond,15.numBytesInRemotePerSecond,88.checkpointAlignmentTime,22.numBytesOut,11.numBytesOut,122.checkpointAlignmentTime,80.currentLowWatermark,84.numRecordsInPerSecond,58.Sink__bad_records.numRecordsInPerSecond,107.buffers.inputQueueLength,67.numBytesInLocalPerSecond,28.buffers.inputQueueLength,10.lastCheckpointSize,15.buffers.outputQueueLength,131.numBytesOut,82.Sink__bad_records.numRecordsInPerSecond,19.numBytesOut,88.numBytesInLocal,35.numRecordsIn,39.checkpointAlignmentTime,66.Sink__bad_records.numRecordsIn,28.buffers.outPoolUsage,5.Sink__bad_records.numRecordsOutPerSecond,19.numBytesOutPerSecond,45.numRecordsInPerSecond,31.buffers.outputQueueLength,25.lastCheckpointSize,121.lastCheckpointSize,14.buffers.outPoolUsage,12.numBytesOutPerSecond,2.numRecordsInPerSecond,67.Sink__bad_records.numRecordsIn,2.numBytesInRemote,26.numBytesOutPerSecond,111.buffers.inPoolUsage,40.checkpointAlignmentTime,33.buffers.inPoolUsage,49.Sink__bad_records.numRecordsOutPerSecond,131.numBytesInLocalPerSecond,131.buffers.inputQueueLength,16.buffers.outPoolUsage,63.numRecordsOutPerSecond,67.numRecordsOut,20.buffers.outputQueueLength,117.Sink__bad_records.numRecordsIn,30.numRecordsInPerSecond,110.Sink__bad_records.numRecordsOut,41.currentLowWatermark,31.numBytesOutPerSecond,140.lastCheckpointSize,99.lastCheckpointSize,101.numBytesInRemote,105.Sink__bad_records.numRecordsOut,53.numBytesInLocalPerSecond,35.Sink__bad_records.numRecordsOut,56.lastCheckpointSize,67.checkpointAlignmentTime,131.buffers.inPoolUsage,86.Sink__bad_records.numRecordsInPerSecond,92.buffers.inPoolUsage,26.numBytesInLocal,90.numRecordsIn,18.buffers.outPoolUsage,74.buffers.inputQueueLength,50.Sink__bad_records.latency,13.buffers.inPoolUsage,63.buffers.inputQueueLength,115.numRecordsOutPerSecond,61.currentLowWatermark,67.lastCheckpointSize,121.numBytesInRemote,78.lastCheckpointSize,42.Sink__bad_records.latency,143.numRecordsInPerSecond,69.numRecordsInPerSecond,77.Sink__bad_records.numRecordsInPerSecond,35.Sink__bad_records.latency,7.numBytesInLocal,8.numBytesInRemotePerSecond,129.buffers.outputQueueLength,73.buffers.inPoolUsage,27.Sink__bad_records.latency,68.numRecordsOut,72.buffers.inPoolUsage,113.numRecordsInPerSecond,40.Sink__bad_records.numRecordsOut,88.lastCheckpointSize,99.Sink__bad_records.numRecordsOutPerSecond,24.numRecordsOutPerSecond,106.numBytesOut,137.numBytesInLocalPerSecond,141.numBytesInRemote,79.numBytesInRemotePerSecond,114.numBytesInRemotePerSecond,118.numRecordsOutPerSecond,42.numBytesInLocalPerSecond,80.numRecordsIn,134.numRecordsOut,56.Sink__bad_records.numRecordsOutPerSecond,39.numRecordsInPerSecond,120.Sink__bad_records.numRecordsIn,50.Sink__bad_records.numRecordsInPerSecond,5.numBytesOut,0.lastCheckpointSize,38.Sink__bad_records.numRecordsInPerSecond,84.numBytesInRemotePerSecond,7.numRecordsOut,53.buffers.inPoolUsage,58.checkpointAlignmentTime,52.buffers.inPoolUsage,80.numRecordsOutPerSecond,141.numRecordsIn,22.currentLowWatermark,48.numBytesInLocalPerSecond,41.buffers.outPoolUsage,8.numRecordsOut,3.Sink__bad_records.numRecordsOutPerSecond,21.currentLowWatermark,119.Sink__bad_records.numRecordsOutPerSecond,17.Sink__bad_records.numRecordsIn,57.lastCheckpointSize,66.buffers.outputQueueLength,50.numRecordsOut,100.numRecordsIn,122.numBytesInRemotePerSecond,85.buffers.inputQueueLength,77.Sink__bad_records.numRecordsOutPerSecond,96.Sink__bad_records.latency,124.Sink__bad_records.numRecordsIn,59.numBytesInLocal,64.numBytesInLocalPerSecond,139.buffers.outPoolUsage,116.Sink__bad_records.latency,143.Sink__bad_records.numRecordsInPerSecond,79.numBytesInRemote,141.buffers.outputQueueLength,93.numBytesInLocal,87.numBytesInRemotePerSecond,132.numRecordsOutPerSecond,88.numRecordsOut,18.numBytesInLocalPerSecond,81.numBytesInLocal,26.buffers.outPoolUsage,22.buffers.outPoolUsage,36.numBytesInRemotePerSecond,21.numBytesOutPerSecond,134.numRecordsIn,21.numRecordsOutPerSecond,107.numRecordsInPerSecond,99.numBytesInRemote,115.Sink__bad_records.numRecordsOut,98.numBytesOutPerSecond,78.Sink__bad_records.latency,120.numBytesInLocalPerSecond,69.buffers.inputQueueLength,120.buffers.inputQueueLength,14.buffers.inPoolUsage,41.numBytesInLocal,5.numBytesInRemotePerSecond,133.Sink__bad_records.numRecordsInPerSecond,87.numRecordsOut,58.numBytesInRemote,133.numRecordsOut,73.Sink__bad_records.numRecordsIn,48.buffers.outputQueueLength,62.numBytesInLocal,66.lastCheckpointSize,68.lastCheckpointSize,130.buffers.inPoolUsage,120.Sink__bad_records.numRecordsOut,60.currentLowWatermark,38.numBytesInRemote,77.lastCheckpointSize,63.Sink__bad_records.numRecordsInPerSecond,69.numRecordsOutPerSecond,91.buffers.inPoolUsage,4.numBytesInLocalPerSecond,74.numBytesInLocal,38.Sink__bad_records.latency,111.checkpointAlignmentTime,27.Sink__bad_records.numRecordsOutPerSecond,55.lastCheckpointSize,30.Sink__bad_records.numRecordsIn,2.checkpointAlignmentTime,29.Sink__bad_records.numRecordsInPerSecond,66.numRecordsOutPerSecond,13.lastCheckpointSize,96.Sink__bad_records.numRecordsOut,50.Sink__bad_records.numRecordsOut,38.numBytesInLocal,58.buffers.inputQueueLength,12.buffers.outputQueueLength,29.numBytesInLocalPerSecond,137.numRecordsInPerSecond,46.lastCheckpointSize,114.numRecordsInPerSecond,109.numBytesOut,99.Sink__bad_records.numRecordsIn,44.numBytesInRemotePerSecond,83.numRecordsIn,65.checkpointAlignmentTime,14.numBytesOutPerSecond,108.Sink__bad_records.latency,6.numRecordsIn,45.Sink__bad_records.numRecordsOut,122.numRecordsIn,78.Sink__bad_records.numRecordsOutPerSecond,35.lastCheckpointSize,108.numBytesInLocal,2.numBytesOut,74.checkpointAlignmentTime,10.Sink__bad_records.numRecordsOutPerSecond,111.numBytesInLocal,129.numBytesInLocal,120.Sink__bad_records.latency,112.numRecordsIn,61.numBytesInRemote,37.numBytesInLocalPerSecond,24.lastCheckpointSize,132.numBytesInLocal,18.numBytesInRemotePerSecond,120.Sink__bad_records.numRecordsInPerSecond,108.Sink__bad_records.numRecordsInPerSecond,136.checkpointAlignmentTime,82.numBytesOut,118.numBytesInLocalPerSecond,51.numRecordsIn,85.numBytesOut,33.buffers.outPoolUsage,125.Sink__bad_records.numRecordsOutPerSecond,90.buffers.inPoolUsage,127.numBytesInLocal,120.checkpointAlignmentTime,12.checkpointAlignmentTime,98.numRecordsIn,32.buffers.outPoolUsage,20.numBytesInLocalPerSecond,106.numRecordsInPerSecond,80.numRecordsOut,88.numBytesOut,93.buffers.inputQueueLength,130.Sink__bad_records.numRecordsOut,86.numRecordsOutPerSecond,10.Sink__bad_records.latency,90.numRecordsOut,21.numBytesInRemote,80.buffers.inPoolUsage,26.numBytesInRemote,77.numBytesOut,24.Sink__bad_records.latency,71.numBytesOut,96.buffers.inputQueueLength,138.checkpointAlignmentTime,93.numBytesOut,31.buffers.outPoolUsage,34.Sink__bad_records.numRecordsIn,74.numBytesOut,66.numBytesInRemote,30.buffers.outPoolUsage,102.Sink__bad_records.numRecordsOutPerSecond,130.numBytesInLocal,11.Sink__bad_records.numRecordsOutPerSecond,101.buffers.inputQueueLength,118.Sink__bad_records.numRecordsOutPerSecond,139.numBytesInRemotePerSecond,9.buffers.inputQueueLength,46.numBytesInRemote,115.numRecordsIn,95.numBytesInLocal,108.checkpointAlignmentTime,79.lastCheckpointSize,91.Sink__bad_records.numRecordsOutPerSecond,118.Sink__bad_records.numRecordsInPerSecond,118.buffers.outputQueueLength,23.numRecordsInPerSecond,63.numBytesOut,15.numBytesOutPerSecond,55.Sink__bad_records.numRecordsOutPerSecond,136.numRecordsInPerSecond,32.numRecordsOutPerSecond,21.Sink__bad_records.numRecordsInPerSecond,60.Sink__bad_records.numRecordsOut,60.numBytesInLocal,98.numRecordsOut,41.numBytesInRemote,46.numRecordsInPerSecond,70.numRecordsOut,80.Sink__bad_records.numRecordsIn,5.numBytesInLocal,6.numBytesInRemote,15.buffers.inPoolUsage,138.numBytesOutPerSecond,96.numBytesOut,71.buffers.inPoolUsage,125.Sink__bad_records.numRecordsOut,42.checkpointAlignmentTime,140.numBytesInRemotePerSecond,26.numBytesInLocalPerSecond,129.numBytesInLocalPerSecond,55.Sink__bad_records.numRecordsOut,89.numRecordsOutPerSecond,57.numBytesInLocal,54.numRecordsIn,115.numBytesInLocalPerSecond,24.numBytesInLocal,90.numRecordsInPerSecond,76.numBytesInRemotePerSecond,34.numBytesInLocalPerSecond,96.Sink__bad_records.numRecordsInPerSecond,27.numRecordsOut,107.numBytesInLocalPerSecond,125.numBytesInRemotePerSecond,73.Sink__bad_records.numRecordsInPerSecond,67.Sink__bad_records.numRecordsInPerSecond,62.Sink__bad_records.numRecordsOutPerSecond,12.numBytesInLocalPerSecond,1.Sink__bad_records.numRecordsIn,40.currentLowWatermark,9.numRecordsIn,111.numBytesInRemotePerSecond,34.buffers.inPoolUsage,15.numBytesInLocalPerSecond,123.numRecordsOut,18.numBytesInRemote,105.Sink__bad_records.latency,25.Sink__bad_records.numRecordsInPerSecond,143.checkpointAlignmentTime,99.numRecordsInPerSecond,92.buffers.outputQueueLength,25.buffers.outPoolUsage,0.numRecordsOut,34.lastCheckpointSize,35.checkpointAlignmentTime,101.buffers.outputQueueLength,126.numBytesInLocalPerSecond,131.Sink__bad_records.latency,50.currentLowWatermark,110.buffers.inPoolUsage,24.buffers.inPoolUsage,4.checkpointAlignmentTime,113.numBytesInLocal,31.numBytesInLocalPerSecond,73.numBytesInRemotePerSecond,3.numRecordsInPerSecond,119.Sink__bad_records.latency,22.numBytesOutPerSecond,1.buffers.inPoolUsage,60.numRecordsInPerSecond,0.Sink__bad_records.latency,65.Sink__bad_records.numRecordsOut,10.numBytesInLocal,63.buffers.outputQueueLength,142.numBytesInLocalPerSecond,120.buffers.inPoolUsage,89.lastCheckpointSize,120.numRecordsInPerSecond,37.buffers.outputQueueLength,81.buffers.inPoolUsage,37.numRecordsOut,113.checkpointAlignmentTime,113.Sink__bad_records.numRecordsIn,4.Sink__bad_records.numRecordsOutPerSecond,47.numBytesInRemotePerSecond,49.Sink__bad_records.numRecordsIn,88.buffers.inputQueueLength,16.numBytesOutPerSecond,35.numRecordsOutPerSecond,99.numBytesOutPerSecond,23.numBytesOutPerSecond,84.Sink__bad_records.numRecordsIn,93.Sink__bad_records.latency,24.buffers.outPoolUsage,60.numBytesOut,141.lastCheckpointSize,72.checkpointAlignmentTime,48.Sink__bad_records.numRecordsInPerSecond,94.numBytesInRemote,83.numRecordsOutPerSecond,137.numRecordsIn,12.currentLowWatermark,47.numRecordsOut,62.buffers.inPoolUsage,99.buffers.inputQueueLength,76.numRecordsIn,70.Sink__bad_records.numRecordsOut,9.Sink__bad_records.numRecordsIn,23.buffers.outputQueueLength,45.lastCheckpointSize,103.numRecordsOut,139.numBytesOutPerSecond,99.numBytesOut,53.Sink__bad_records.latency,43.numBytesInLocal,113.numRecordsOut,112.numRecordsOutPerSecond,67.Sink__bad_records.latency,114.Sink__bad_records.numRecordsInPerSecond,38.numRecordsOutPerSecond,84.Sink__bad_records.numRecordsOutPerSecond,5.Sink__bad_records.numRecordsIn,31.currentLowWatermark,12.lastCheckpointSize,126.numBytesInRemote,137.Sink__bad_records.numRecordsInPerSecond,130.lastCheckpointSize,41.Sink__bad_records.numRecordsIn,53.numRecordsInPerSecond,33.numBytesInRemotePerSecond,23.numBytesInLocalPerSecond,140.Sink__bad_records.numRecordsOut,25.numRecordsIn,135.Sink__bad_records.numRecordsOut,23.lastCheckpointSize,43.buffers.inPoolUsage,76.numRecordsInPerSecond,131.checkpointAlignmentTime,52.numRecordsInPerSecond,70.numBytesInLocal,56.numBytesInLocalPerSecond,3.lastCheckpointSize,42.numBytesInRemote,71.numBytesInRemotePerSecond,101.buffers.inPoolUsage,36.numRecordsIn,8.buffers.outputQueueLength,131.Sink__bad_records.numRecordsOut,129.numRecordsOutPerSecond,48.numRecordsIn,70.checkpointAlignmentTime,77.buffers.inPoolUsage,94.Sink__bad_records.numRecordsOut,65.numRecordsInPerSecond,45.Sink__bad_records.numRecordsIn,67.numBytesInLocal,17.numRecordsInPerSecond,88.Sink__bad_records.numRecordsIn,134.numBytesInLocalPerSecond,77.numBytesInLocalPerSecond,71.currentLowWatermark,106.buffers.outputQueueLength,8.Sink__bad_records.numRecordsInPerSecond,107.numRecordsOut,86.buffers.outputQueueLength,57.numRecordsOut,0.numBytesOutPerSecond,100.Sink__bad_records.latency,124.Sink__bad_records.numRecordsOutPerSecond,8.Sink__bad_records.numRecordsOutPerSecond,129.Sink__bad_records.latency,62.checkpointAlignmentTime,140.numBytesInLocal,94.numRecordsOut,48.numBytesInRemotePerSecond,98.numBytesInLocalPerSecond,110.Sink__bad_records.numRecordsOutPerSecond,11.buffers.outPoolUsage,30.numBytesOutPerSecond,50.buffers.inputQueueLength,12.numRecordsIn,101.Sink__bad_records.numRecordsIn,138.Sink__bad_records.numRecordsIn,123.checkpointAlignmentTime,97.checkpointAlignmentTime,24.numRecordsIn,93.buffers.outPoolUsage,33.Sink__bad_records.numRecordsOutPerSecond,143.Sink__bad_records.latency,6.numRecordsOutPerSecond,90.currentLowWatermark,128.Sink__bad_records.numRecordsInPerSecond,81.numBytesOutPerSecond,9.buffers.inPoolUsage,89.checkpointAlignmentTime,137.numBytesInLocal,116.buffers.outPoolUsage,120.buffers.outputQueueLength,27.numBytesOut,56.Sink__bad_records.numRecordsOut,134.numBytesInRemotePerSecond,44.numBytesOutPerSecond,140.Sink__bad_records.numRecordsIn,74.numRecordsOutPerSecond,131.Sink__bad_records.numRecordsOutPerSecond,90.Sink__bad_records.numRecordsIn,110.currentLowWatermark,113.numBytesInLocalPerSecond,92.buffers.inputQueueLength,57.numBytesInRemotePerSecond,143.numBytesInRemotePerSecond,12.numBytesOut,122.numBytesInLocal,23.numBytesOut,26.Sink__bad_records.numRecordsOutPerSecond,115.checkpointAlignmentTime,14.numBytesInLocalPerSecond,109.numRecordsOut,59.currentLowWatermark,116.numRecordsInPerSecond,115.buffers.outputQueueLength,16.numBytesOut,78.numRecordsInPerSecond,111.numRecordsOutPerSecond,96.numRecordsOut,14.Sink__bad_records.latency,12.Sink__bad_records.numRecordsOutPerSecond,68.numRecordsIn,62.buffers.outPoolUsage,103.numBytesInRemote,41.numBytesOut,18.numRecordsOut,40.numBytesInLocalPerSecond,52.Sink__bad_records.numRecordsInPerSecond,32.numRecordsIn,48.Sink__bad_records.latency,43.Sink__bad_records.numRecordsOut,133.lastCheckpointSize,49.numBytesOut,20.numRecordsIn,28.numRecordsOutPerSecond,86.lastCheckpointSize,77.buffers.outputQueueLength,140.numRecordsInPerSecond,138.Sink__bad_records.latency,89.numBytesInRemote,70.numBytesInRemote,122.buffers.outPoolUsage,129.numBytesInRemotePerSecond,45.numBytesOut,80.numBytesInRemotePerSecond,89.Sink__bad_records.numRecordsInPerSecond,38.numBytesOutPerSecond,100.numBytesInRemotePerSecond,34.numBytesOut,13.numBytesInRemote,14.numBytesInRemotePerSecond,30.numBytesOut,52.numBytesOut,110.Sink__bad_records.numRecordsInPerSecond,19.Sink__bad_records.numRecordsInPerSecond,80.numBytesInLocalPerSecond,10.numRecordsOutPerSecond,129.numRecordsInPerSecond,39.Sink__bad_records.numRecordsIn,52.buffers.outputQueueLength,134.Sink__bad_records.latency,45.buffers.inputQueueLength,95.buffers.outputQueueLength,94.Sink__bad_records.numRecordsIn,71.numBytesInRemote,61.numBytesInLocalPerSecond,61.Sink__bad_records.numRecordsOutPerSecond,7.numBytesInRemote,19.Sink__bad_records.numRecordsOutPerSecond,62.numBytesInRemotePerSecond,119.numRecordsIn,19.Sink__bad_records.numRecordsOut,12.Sink__bad_records.numRecordsIn,10.numBytesInRemotePerSecond,46.checkpointAlignmentTime,28.numRecordsIn,66.buffers.inputQueueLength,131.numBytesInRemote,47.Sink__bad_records.numRecordsOutPerSecond,80.Sink__bad_records.numRecordsOut,51.Sink__bad_records.numRecordsIn,143.numRecordsOutPerSecond,62.Sink__bad_records.latency,88.Sink__bad_records.numRecordsInPerSecond,34.buffers.outputQueueLength,131.lastCheckpointSize,103.Sink__bad_records.numRecordsOutPerSecond,136.buffers.outPoolUsage,42.lastCheckpointSize,128.Sink__bad_records.numRecordsIn,26.buffers.inputQueueLength,24.numBytesOutPerSecond,19.buffers.outPoolUsage,140.numRecordsOut,52.numRecordsIn,108.Sink__bad_records.numRecordsOut,80.Sink__bad_records.latency,55.Sink__bad_records.numRecordsIn,55.numRecordsOutPerSecond,105.Sink__bad_records.numRecordsIn,38.numBytesOut,68.Sink__bad_records.numRecordsOutPerSecond,141.numBytesOutPerSecond,8.numBytesOutPerSecond,66.Sink__bad_records.latency,39.lastCheckpointSize,37.lastCheckpointSize,78.Sink__bad_records.numRecordsIn,18.Sink__bad_records.numRecordsInPerSecond,53.Sink__bad_records.numRecordsInPerSecond,128.lastCheckpointSize,40.Sink__bad_records.numRecordsOutPerSecond,108.numBytesInLocalPerSecond,1.Sink__bad_records.numRecordsOutPerSecond,142.numRecordsOut,3.currentLowWatermark,36.numRecordsOutPerSecond,81.numBytesOut,68.buffers.inputQueueLength,17.Sink__bad_records.numRecordsInPerSecond,127.numRecordsIn,139.buffers.inputQueueLength,90.buffers.inputQueueLength,129.Sink__bad_records.numRecordsInPerSecond,32.Sink__bad_records.latency,44.lastCheckpointSize,130.numRecordsOut,47.buffers.inputQueueLength,69.numBytesOutPerSecond,89.numRecordsInPerSecond,63.numRecordsInPerSecond,78.numBytesOut,89.numBytesOut,91.lastCheckpointSize,39.Sink__bad_records.latency,118.buffers.inputQueueLength,92.numBytesOut,58.lastCheckpointSize,15.numRecordsInPerSecond,54.Sink__bad_records.numRecordsInPerSecond,132.Sink__bad_records.numRecordsOut,70.numBytesOut,34.Sink__bad_records.numRecordsInPerSecond,136.currentLowWatermark,3.numRecordsOut,103.numRecordsIn,38.buffers.outputQueueLength,16.numRecordsOut,61.Sink__bad_records.numRecordsIn,6.buffers.outPoolUsage,129.numBytesOutPerSecond,68.numRecordsOutPerSecond,97.currentLowWatermark,54.numRecordsInPerSecond,75.numBytesInLocalPerSecond,9.Sink__bad_records.numRecordsInPerSecond,89.buffers.inputQueueLength,26.checkpointAlignmentTime,2.buffers.inPoolUsage,102.buffers.outputQueueLength,50.numRecordsInPerSecond,61.buffers.outputQueueLength,132.numBytesInRemote,96.numBytesInLocalPerSecond,103.checkpointAlignmentTime,54.numBytesInLocalPerSecond,5.lastCheckpointSize,135.numRecordsOutPerSecond,76.buffers.outPoolUsage,39.buffers.inPoolUsage,50.numRecordsOutPerSecond,91.buffers.outputQueueLength,51.buffers.inPoolUsage,1.lastCheckpointSize,15.numRecordsOutPerSecond,104.buffers.inputQueueLength,78.currentLowWatermark,88.numRecordsIn,7.Sink__bad_records.numRecordsInPerSecond,30.Sink__bad_records.numRecordsOut,84.lastCheckpointSize,16.buffers.inPoolUsage,82.numRecordsOutPerSecond,2.numBytesInLocal,14.numRecordsOut,134.numBytesOut,50.checkpointAlignmentTime,16.Sink__bad_records.numRecordsIn,84.numRecordsIn,67.numBytesOut,56.numBytesOut,57.Sink__bad_records.numRecordsOut,121.numBytesOutPerSecond,29.Sink__bad_records.numRecordsIn,27.numBytesOutPerSecond,70.buffers.inPoolUsage,59.buffers.outputQueueLength,36.buffers.outPoolUsage,96.Sink__bad_records.numRecordsOutPerSecond,18.numBytesOutPerSecond,14.numBytesInRemote,55.numBytesOutPerSecond,6.checkpointAlignmentTime,38.checkpointAlignmentTime,16.numBytesInLocalPerSecond,105.numRecordsInPerSecond,13.numBytesOutPerSecond,116.numBytesInRemotePerSecond,130.Sink__bad_records.numRecordsIn,133.buffers.outputQueueLength,101.numRecordsInPerSecond,35.buffers.inPoolUsage,75.numBytesInRemotePerSecond,69.checkpointAlignmentTime,64.numRecordsIn,95.numBytesOutPerSecond,68.buffers.outputQueueLength,117.currentLowWatermark,41.Sink__bad_records.latency,135.numBytesOutPerSecond,5.numRecordsOut,110.lastCheckpointSize,36.Sink__bad_records.numRecordsInPerSecond,87.numRecordsOutPerSecond,112.buffers.inPoolUsage,42.buffers.outPoolUsage,127.Sink__bad_records.numRecordsInPerSecond,102.buffers.inputQueueLength,134.Sink__bad_records.numRecordsIn,29.buffers.outputQueueLength,104.numBytesInRemote,102.buffers.outPoolUsage,44.numRecordsIn,107.numBytesInRemotePerSecond,32.buffers.inPoolUsage,54.buffers.inPoolUsage,58.Sink__bad_records.numRecordsOut,22.Sink__bad_records.numRecordsIn,31.buffers.inputQueueLength,2.numBytesInLocalPerSecond,58.buffers.inPoolUsage,107.numRecordsIn,82.checkpointAlignmentTime,2.numBytesInRemotePerSecond,115.numBytesOutPerSecond,81.Sink__bad_records.numRecordsOut,130.numRecordsOutPerSecond,52.buffers.inputQueueLength,37.Sink__bad_records.numRecordsInPerSecond,63.Sink__bad_records.numRecordsOutPerSecond,141.numBytesInLocalPerSecond,52.currentLowWatermark,116.numRecordsOutPerSecond,39.numBytesInRemotePerSecond,123.buffers.inputQueueLength,43.numBytesInRemote,61.lastCheckpointSize,71.buffers.inputQueueLength,35.numBytesInRemotePerSecond,68.Sink__bad_records.numRecordsIn,116.buffers.inPoolUsage,65.lastCheckpointSize,65.Sink__bad_records.numRecordsIn,87.buffers.inputQueueLength,49.numRecordsOutPerSecond,139.buffers.inPoolUsage,101.numBytesOutPerSecond,88.numBytesInRemote,33.currentLowWatermark,109.Sink__bad_records.numRecordsOut,142.buffers.inputQueueLength,2.Sink__bad_records.latency,3.numBytesOutPerSecond,35.Sink__bad_records.numRecordsInPerSecond,133.Sink__bad_records.numRecordsOut,1.numRecordsOut,21.buffers.outputQueueLength,126.Sink__bad_records.numRecordsInPerSecond,10.currentLowWatermark,75.Sink__bad_records.latency,95.Sink__bad_records.numRecordsOut,9.numBytesInLocal,103.numRecordsInPerSecond,72.numBytesInRemote,107.lastCheckpointSize,89.numBytesOutPerSecond,14.lastCheckpointSize,14.currentLowWatermark,41.numBytesInRemotePerSecond,61.numRecordsInPerSecond,8.numBytesInRemote,56.buffers.outPoolUsage,18.lastCheckpointSize,142.numRecordsInPerSecond,44.Sink__bad_records.numRecordsOut,96.buffers.inPoolUsage,9.numRecordsInPerSecond,113.Sink__bad_records.latency,75.numBytesOutPerSecond,25.buffers.outputQueueLength,117.Sink__bad_records.numRecordsOutPerSecond,35.numBytesInLocalPerSecond,61.numBytesOutPerSecond,59.numBytesInRemote,31.numRecordsOutPerSecond,107.Sink__bad_records.latency,71.Sink__bad_records.latency,106.numBytesInLocalPerSecond,33.numBytesOutPerSecond,54.Sink__bad_records.numRecordsOutPerSecond,21.lastCheckpointSize,135.buffers.inPoolUsage,6.Sink__bad_records.latency,133.Sink__bad_records.numRecordsOutPerSecond,128.currentLowWatermark,17.Sink__bad_records.numRecordsOutPerSecond,131.numRecordsIn,86.numBytesOutPerSecond,5.numBytesInRemote,87.numBytesInRemote,101.Sink__bad_records.numRecordsOutPerSecond,14.numBytesInLocal,95.numRecordsOutPerSecond,119.checkpointAlignmentTime,64.buffers.outputQueueLength,54.Sink__bad_records.numRecordsOut,55.Sink__bad_records.numRecordsInPerSecond,113.numBytesInRemotePerSecond,13.buffers.outputQueueLength,128.buffers.outputQueueLength,53.buffers.outPoolUsage,121.Sink__bad_records.numRecordsIn,10.numRecordsOut,106.buffers.inputQueueLength,136.buffers.outputQueueLength,34.numRecordsOutPerSecond,84.numBytesOutPerSecond,26.numRecordsInPerSecond,59.buffers.inputQueueLength,15.checkpointAlignmentTime,47.buffers.inPoolUsage,63.numBytesInLocalPerSecond,36.currentLowWatermark,140.Sink__bad_records.numRecordsOutPerSecond,29.numBytesInLocal,15.numBytesInRemote,51.buffers.outPoolUsage,31.checkpointAlignmentTime,142.numRecordsIn,13.numRecordsInPerSecond,100.buffers.inputQueueLength,70.Sink__bad_records.numRecordsIn,91.numRecordsInPerSecond,137.buffers.inputQueueLength,126.Sink__bad_records.numRecordsOutPerSecond,12.numBytesInRemote,86.numRecordsOut,3.numRecordsIn,109.numBytesOutPerSecond,35.Sink__bad_records.numRecordsOutPerSecond,92.Sink__bad_records.numRecordsOut,7.numRecordsIn,115.Sink__bad_records.numRecordsOutPerSecond,105.buffers.inPoolUsage,142.buffers.outputQueueLength,26.Sink__bad_records.numRecordsIn,89.currentLowWatermark,72.Sink__bad_records.numRecordsInPerSecond,56.buffers.outputQueueLength,112.numBytesInRemotePerSecond,112.numRecordsInPerSecond,17.buffers.inputQueueLength,92.numBytesInRemotePerSecond,31.Sink__bad_records.numRecordsOut,55.currentLowWatermark,133.numBytesInRemote,98.numRecordsOutPerSecond,121.numBytesInRemotePerSecond,65.buffers.outputQueueLength,16.Sink__bad_records.numRecordsInPerSecond,73.buffers.outputQueueLength,43.numRecordsInPerSecond,28.numBytesInLocalPerSecond,41.numBytesOutPerSecond,28.numRecordsOut,117.numBytesInLocalPerSecond,127.numBytesInLocalPerSecond,113.currentLowWatermark,9.numRecordsOut,69.numBytesInRemotePerSecond,74.numRecordsInPerSecond,69.numRecordsOut,119.Sink__bad_records.numRecordsOut,49.checkpointAlignmentTime,110.numRecordsIn,7.numBytesOut,42.Sink__bad_records.numRecordsOutPerSecond,136.numRecordsOut,100.numRecordsOutPerSecond,6.Sink__bad_records.numRecordsInPerSecond,59.Sink__bad_records.numRecordsIn,12.buffers.inPoolUsage,22.buffers.outputQueueLength,105.Sink__bad_records.numRecordsOutPerSecond,87.numBytesInLocal,52.numBytesInLocalPerSecond,109.Sink__bad_records.numRecordsIn,119.buffers.outputQueueLength,9.numBytesInRemotePerSecond,99.buffers.outputQueueLength,26.Sink__bad_records.latency,70.buffers.outputQueueLength,93.checkpointAlignmentTime,90.numBytesInLocal,123.numBytesOut,112.numBytesOut,22.checkpointAlignmentTime,74.Sink__bad_records.numRecordsIn,46.buffers.inPoolUsage,132.buffers.inPoolUsage,40.lastCheckpointSize,94.buffers.inputQueueLength,44.numRecordsOutPerSecond,82.Sink__bad_records.numRecordsOut,119.buffers.outPoolUsage,47.numBytesInLocal,132.numBytesInLocalPerSecond,85.numRecordsOutPerSecond,92.Sink__bad_records.latency,104.numBytesInRemotePerSecond,109.currentLowWatermark,79.buffers.outPoolUsage,118.checkpointAlignmentTime,103.numRecordsOutPerSecond,137.buffers.outputQueueLength,20.numBytesInRemote,92.numRecordsIn,122.Sink__bad_records.numRecordsOutPerSecond,112.Sink__bad_records.numRecordsOutPerSecond,71.numRecordsIn,88.Sink__bad_records.latency,30.checkpointAlignmentTime,33.buffers.inputQueueLength,41.Sink__bad_records.numRecordsOut,74.buffers.inPoolUsage,31.Sink__bad_records.numRecordsOutPerSecond,104.buffers.inPoolUsage,132.numRecordsOut,59.Sink__bad_records.numRecordsOut,50.numBytesInLocal,65.numRecordsOut,65.numBytesInLocal,17.currentLowWatermark,5.numBytesOutPerSecond,125.Sink__bad_records.numRecordsIn,20.Sink__bad_records.numRecordsInPerSecond,104.Sink__bad_records.latency,35.numBytesInRemote,32.numBytesInLocal,102.numRecordsIn,40.buffers.inPoolUsage,80.numRecordsInPerSecond,130.numBytesOutPerSecond,134.Sink__bad_records.numRecordsOut,117.numBytesInLocal,135.lastCheckpointSize,77.checkpointAlignmentTime,109.Sink__bad_records.numRecordsInPerSecond,105.numBytesInRemote,130.Sink__bad_records.latency,27.buffers.inPoolUsage,125.buffers.outPoolUsage,138.numRecordsInPerSecond,47.buffers.outPoolUsage,22.buffers.inputQueueLength,101.numBytesOut,35.numBytesOutPerSecond,5.numRecordsInPerSecond,51.lastCheckpointSize,9.Sink__bad_records.latency,142.Sink__bad_records.numRecordsInPerSecond,93.buffers.inPoolUsage,127.checkpointAlignmentTime,136.numBytesInLocalPerSecond,56.numRecordsInPerSecond,16.buffers.outputQueueLength,28.Sink__bad_records.numRecordsOutPerSecond,98.Sink__bad_records.numRecordsOutPerSecond,112.Sink__bad_records.latency,31.Sink__bad_records.numRecordsIn,102.numBytesInLocal,143.numRecordsIn,33.numBytesInLocalPerSecond,30.buffers.outputQueueLength,127.numBytesOutPerSecond,21.numBytesInLocalPerSecond,67.numBytesInRemote,96.buffers.outPoolUsage,70.Sink__bad_records.numRecordsOutPerSecond,135.numBytesInLocal,120.numBytesInLocal,26.numBytesInRemotePerSecond,124.lastCheckpointSize,87.Sink__bad_records.numRecordsInPerSecond,119.lastCheckpointSize,40.numBytesInRemotePerSecond,32.numBytesInRemotePerSecond,139.numRecordsIn,19.numRecordsInPerSecond,80.numBytesOut,81.Sink__bad_records.numRecordsIn,10.numBytesOutPerSecond,29.numBytesOutPerSecond,44.numBytesInLocalPerSecond,86.checkpointAlignmentTime,35.Sink__bad_records.numRecordsIn,123.buffers.inPoolUsage,93.numBytesInRemote,98.Sink__bad_records.numRecordsIn,30.lastCheckpointSize,1.numRecordsOutPerSecond,106.Sink__bad_records.numRecordsOut,110.Sink__bad_records.numRecordsIn,23.numBytesInRemotePerSecond,54.Sink__bad_records.latency,113.Sink__bad_records.numRecordsInPerSecond,69.Sink__bad_records.numRecordsOut,83.Sink__bad_records.latency,124.numBytesOutPerSecond,63.currentLowWatermark,102.numBytesInRemote,17.Sink__bad_records.latency,91.numBytesOut,127.numBytesInRemote,50.numBytesOutPerSecond,67.numRecordsInPerSecond,47.numRecordsOutPerSecond,9.numBytesInLocalPerSecond,85.numRecordsInPerSecond,138.numBytesInRemotePerSecond,87.buffers.outPoolUsage,0.numBytesInLocalPerSecond,6.numBytesInLocalPerSecond,61.numRecordsOutPerSecond,127.numRecordsOutPerSecond,111.numRecordsOut,75.buffers.inputQueueLength,72.numRecordsOut,125.buffers.inputQueueLength,65.buffers.inPoolUsage,97.numBytesInRemotePerSecond,21.buffers.inPoolUsage,63.numRecordsIn,91.Sink__bad_records.numRecordsInPerSecond,9.buffers.outPoolUsage,84.buffers.inPoolUsage,75.Sink__bad_records.numRecordsOutPerSecond,66.numBytesInRemotePerSecond,40.Sink__bad_records.latency,39.buffers.outPoolUsage,51.numRecordsOut,58.numBytesInLocalPerSecond,125.numBytesInRemote,85.buffers.inPoolUsage,78.checkpointAlignmentTime,51.Sink__bad_records.numRecordsInPerSecond,80.buffers.inputQueueLength,90.checkpointAlignmentTime,32.numRecordsInPerSecond,133.buffers.outPoolUsage,97.Sink__bad_records.latency,142.buffers.inPoolUsage,23.Sink__bad_records.latency,19.buffers.inPoolUsage,130.numBytesOut,130.numBytesInRemotePerSecond,142.buffers.outPoolUsage,100.numBytesInRemote,124.numRecordsOutPerSecond,26.numRecordsOutPerSecond,60.numRecordsIn,111.Sink__bad_records.numRecordsInPerSecond,84.Sink__bad_records.numRecordsOut,10.numBytesInLocalPerSecond,82.buffers.outputQueueLength,143.buffers.inPoolUsage,95.numBytesInRemote,82.numRecordsOut,126.checkpointAlignmentTime,82.Sink__bad_records.numRecordsOutPerSecond,126.numBytesOutPerSecond,18.checkpointAlignmentTime,83.numBytesInRemotePerSecond,55.numRecordsOut,63.Sink__bad_records.latency,39.numBytesInLocalPerSecond,92.numRecordsOut,130.numBytesInRemote,63.lastCheckpointSize,119.numRecordsOut,109.numRecordsInPerSecond,66.buffers.inPoolUsage,7.buffers.inputQueueLength,50.numBytesInRemote,68.Sink__bad_records.numRecordsInPerSecond,140.currentLowWatermark,121.Sink__bad_records.numRecordsOut,101.lastCheckpointSize,41.numRecordsOut,104.numBytesInLocal,118.numBytesOutPerSecond,31.numRecordsOut,71.numRecordsOutPerSecond,34.numBytesInLocal,129.currentLowWatermark,137.numRecordsOutPerSecond,119.numBytesInLocal,135.numRecordsIn,5.buffers.inPoolUsage,124.buffers.inPoolUsage,85.Sink__bad_records.numRecordsIn,20.buffers.inPoolUsage,65.numBytesInRemote,110.numBytesInRemote,36.numBytesOutPerSecond,16.numBytesInLocal,114.Sink__bad_records.numRecordsIn,141.numRecordsOutPerSecond,45.numRecordsOut,130.buffers.inputQueueLength,16.numRecordsIn,8.Sink__bad_records.numRecordsIn,25.currentLowWatermark,6.numBytesOutPerSecond,18.Sink__bad_records.numRecordsOut,48.Sink__bad_records.numRecordsIn,114.buffers.inputQueueLength,44.currentLowWatermark,74.lastCheckpointSize,40.numBytesInRemote,67.numRecordsIn,103.Sink__bad_records.numRecordsOut,6.Sink__bad_records.numRecordsOutPerSecond,64.buffers.inputQueueLength,47.buffers.outputQueueLength,92.numBytesOutPerSecond,111.buffers.outputQueueLength,127.numRecordsInPerSecond,93.Sink__bad_records.numRecordsOutPerSecond,71.numBytesInLocalPerSecond,111.buffers.inputQueueLength,45.buffers.outPoolUsage,105.numRecordsOut,59.numRecordsOut,135.Sink__bad_records.latency,78.numBytesOutPerSecond,82.numBytesInLocalPerSecond,138.Sink__bad_records.numRecordsOutPerSecond,23.numRecordsOutPerSecond,57.numRecordsOutPerSecond,83.buffers.inputQueueLength,28.buffers.inPoolUsage,112.lastCheckpointSize,46.Sink__bad_records.numRecordsOut,141.numBytesOut,106.numRecordsIn,101.numRecordsOut,118.Sink__bad_records.numRecordsIn,21.numRecordsInPerSecond,102.currentLowWatermark,52.numBytesInLocal,35.numRecordsOut,37.numBytesInRemote,96.numRecordsIn,0.buffers.outputQueueLength,49.numBytesInLocal,49.lastCheckpointSize,97.Sink__bad_records.numRecordsOut,4.numRecordsOutPerSecond,17.numBytesInRemotePerSecond,56.numRecordsIn,49.numRecordsOut,82.currentLowWatermark,85.checkpointAlignmentTime,0.numBytesInLocal,127.buffers.outPoolUsage,70.Sink__bad_records.numRecordsInPerSecond,132.numBytesOutPerSecond,109.lastCheckpointSize,85.numBytesInLocal,129.numRecordsOut,47.numBytesInLocalPerSecond,54.checkpointAlignmentTime,100.checkpointAlignmentTime,121.currentLowWatermark,16.lastCheckpointSize,37.numRecordsInPerSecond,2.Sink__bad_records.numRecordsIn,42.Sink__bad_records.numRecordsIn,24.Sink__bad_records.numRecordsOutPerSecond,57.Sink__bad_records.latency,27.lastCheckpointSize,23.checkpointAlignmentTime,115.numRecordsOut,121.Sink__bad_records.latency,20.numRecordsOutPerSecond,121.buffers.outputQueueLength,56.numBytesInRemotePerSecond,39.numRecordsOut,65.Sink__bad_records.numRecordsOutPerSecond,73.numBytesInLocal,114.checkpointAlignmentTime,33.Sink__bad_records.numRecordsOut,79.numBytesInLocal,45.numRecordsOutPerSecond,87.Sink__bad_records.numRecordsIn,91.buffers.outPoolUsage,35.buffers.outputQueueLength,49.Sink__bad_records.numRecordsInPerSecond,142.numBytesInRemotePerSecond,45.checkpointAlignmentTime,35.buffers.inputQueueLength,85.currentLowWatermark,81.buffers.inputQueueLength,94.buffers.outputQueueLength,96.lastCheckpointSize,46.Sink__bad_records.numRecordsIn,50.Sink__bad_records.numRecordsIn,142.Sink__bad_records.numRecordsOutPerSecond,20.numBytesInRemotePerSecond,52.Sink__bad_records.numRecordsOut,49.numBytesInRemotePerSecond,57.numBytesInRemote,143.numBytesInLocal,143.currentLowWatermark,1.Sink__bad_records.latency,98.buffers.inputQueueLength,66.Sink__bad_records.numRecordsInPerSecond,97.numBytesInRemote,63.numBytesInRemotePerSecond,51.Sink__bad_records.numRecordsOutPerSecond,84.numRecordsOutPerSecond,113.buffers.inputQueueLength,57.buffers.inPoolUsage,137.Sink__bad_records.numRecordsIn,110.buffers.outPoolUsage,126.numRecordsOut,118.currentLowWatermark,47.Sink__bad_records.numRecordsOut,100.Sink__bad_records.numRecordsIn,128.Sink__bad_records.latency,102.numBytesOutPerSecond,26.currentLowWatermark,35.numRecordsInPerSecond,7.buffers.outputQueueLength,119.numBytesInRemotePerSecond,135.Sink__bad_records.numRecordsOutPerSecond,93.Sink__bad_records.numRecordsIn,46.numBytesInLocalPerSecond,8.numBytesInLocalPerSecond,38.buffers.inPoolUsage,72.numBytesInRemotePerSecond,105.buffers.outputQueueLength,70.numRecordsInPerSecond,78.buffers.outputQueueLength,106.Sink__bad_records.numRecordsInPerSecond,83.numBytesOutPerSecond,114.buffers.outputQueueLength,133.numRecordsOutPerSecond,126.numBytesInRemotePerSecond,122.Sink__bad_records.numRecordsOut,2.numBytesOutPerSecond,130.checkpointAlignmentTime,2.buffers.inputQueueLength,33.numRecordsIn,22.Sink__bad_records.latency,103.numBytesInLocalPerSecond,46.numBytesOutPerSecond,28.Sink__bad_records.numRecordsOut,143.Sink__bad_records.numRecordsIn,79.Sink__bad_records.numRecordsOut,22.Sink__bad_records.numRecordsOutPerSecond,114.buffers.outPoolUsage,39.currentLowWatermark,23.numRecordsIn,8.buffers.inPoolUsage,109.buffers.inPoolUsage,9.lastCheckpointSize,24.numBytesInLocalPerSecond,111.numBytesInRemote,140.Sink__bad_records.numRecordsInPerSecond,27.numRecordsIn,100.buffers.outPoolUsage,119.numBytesInLocalPerSecond,72.Sink__bad_records.numRecordsOutPerSecond,113.Sink__bad_records.numRecordsOutPerSecond,114.numRecordsOutPerSecond,116.numBytesOutPerSecond,28.lastCheckpointSize,37.Sink__bad_records.numRecordsOutPerSecond,97.buffers.inputQueueLength,40.numBytesOut,78.buffers.inPoolUsage,134.buffers.inPoolUsage,17.numRecordsIn,140.numBytesInRemote,77.numBytesOutPerSecond,43.numRecordsIn,48.numBytesOut,3.numBytesInRemote,76.buffers.inPoolUsage,132.Sink__bad_records.numRecordsInPerSecond,73.numRecordsOut,117.Sink__bad_records.numRecordsOut,122.Sink__bad_records.numRecordsInPerSecond,95.lastCheckpointSize,10.buffers.inPoolUsage,98.currentLowWatermark,104.buffers.outPoolUsage,9.checkpointAlignmentTime,137.numBytesInRemotePerSecond,102.Sink__bad_records.numRecordsOut,114.Sink__bad_records.numRecordsOutPerSecond,57.buffers.inputQueueLength,64.buffers.outPoolUsage,98.Sink__bad_records.numRecordsOut,31.numBytesOut,1.Sink__bad_records.numRecordsOut,137.currentLowWatermark,51.numBytesOut,74.Sink__bad_records.numRecordsInPerSecond,96.checkpointAlignmentTime,56.numBytesOutPerSecond,92.numBytesInLocalPerSecond,118.buffers.outPoolUsage,11.Sink__bad_records.numRecordsIn,29.Sink__bad_records.numRecordsOutPerSecond,139.numRecordsOutPerSecond,79.currentLowWatermark,71.checkpointAlignmentTime,96.numRecordsOutPerSecond,78.buffers.inputQueueLength,20.numBytesOut,139.lastCheckpointSize,87.Sink__bad_records.numRecordsOutPerSecond,95.buffers.inPoolUsage,43.checkpointAlignmentTime,62.numBytesOut,10.numRecordsInPerSecond,45.currentLowWatermark,19.numRecordsIn,58.buffers.outPoolUsage,139.checkpointAlignmentTime,99.numBytesInRemotePerSecond,10.numBytesInRemote,117.buffers.inPoolUsage,34.numRecordsInPerSecond,63.numBytesOutPerSecond,131.Sink__bad_records.numRecordsInPerSecond,129.Sink__bad_records.numRecordsIn,68.buffers.outPoolUsage,29.numBytesInRemote,124.currentLowWatermark,49.numRecordsIn,115.buffers.inPoolUsage,124.numBytesInLocalPerSecond,108.buffers.outPoolUsage,33.numRecordsInPerSecond,49.numRecordsInPerSecond,73.numBytesOutPerSecond,59.numBytesOut,33.numRecordsOutPerSecond,95.numRecordsInPerSecond,75.Sink__bad_records.numRecordsInPerSecond,52.numBytesOutPerSecond,123.numRecordsInPerSecond,104.Sink__bad_records.numRecordsIn,25.numRecordsOutPerSecond,96.numRecordsInPerSecond,59.buffers.inPoolUsage,28.buffers.outputQueueLength,39.numRecordsIn,70.Sink__bad_records.latency,4.Sink__bad_records.numRecordsInPerSecond,136.buffers.inPoolUsage,103.currentLowWatermark,117.lastCheckpointSize,40.buffers.outputQueueLength,74.numRecordsOut,31.numRecordsIn,58.Sink__bad_records.numRecordsOutPerSecond,21.numRecordsIn,54.buffers.outPoolUsage,47.currentLowWatermark,64.currentLowWatermark,97.buffers.inPoolUsage,58.Sink__bad_records.latency,80.Sink__bad_records.numRecordsOutPerSecond,11.numRecordsIn,24.buffers.outputQueueLength,106.numBytesOutPerSecond,75.numRecordsOut,105.currentLowWatermark,66.currentLowWatermark,50.Sink__bad_records.numRecordsOutPerSecond,30.Sink__bad_records.numRecordsOutPerSecond,83.currentLowWatermark,79.Sink__bad_records.numRecordsIn,122.currentLowWatermark,29.numRecordsIn,54.Sink__bad_records.numRecordsIn,25.numBytesInLocalPerSecond,34.numRecordsOut,60.Sink__bad_records.numRecordsIn,137.lastCheckpointSize,84.numBytesOut,67.numBytesOutPerSecond,18.currentLowWatermark,71.buffers.outPoolUsage,0.numRecordsInPerSecond,141.Sink__bad_records.numRecordsOutPerSecond,124.numBytesInLocal,143.numBytesInLocalPerSecond,8.Sink__bad_records.numRecordsOut,45.numBytesInRemotePerSecond,130.buffers.outPoolUsage,4.buffers.outPoolUsage,53.Sink__bad_records.numRecordsOut,113.numRecordsOutPerSecond,62.numRecordsOut,141.currentLowWatermark,112.numBytesInRemote,70.buffers.outPoolUsage,97.numRecordsIn,99.buffers.inPoolUsage,16.numBytesInRemote,19.numBytesInLocalPerSecond,103.Sink__bad_records.latency,28.currentLowWatermark,122.numBytesInLocalPerSecond,65.numBytesInLocalPerSecond,75.numBytesOut,86.numBytesOut,8.numRecordsOutPerSecond,48.Sink__bad_records.numRecordsOut,123.numBytesOutPerSecond,133.Sink__bad_records.latency,87.currentLowWatermark,38.numBytesInRemotePerSecond,95.numBytesOut,73.numBytesOut,126.currentLowWatermark,51.numBytesInLocalPerSecond,63.numBytesInRemote,24.numRecordsOut,75.numRecordsIn,107.Sink__bad_records.numRecordsInPerSecond,133.numBytesInLocal,73.lastCheckpointSize,49.buffers.inPoolUsage,122.numBytesOutPerSecond,89.numBytesInLocal,98.numBytesInLocal,89.buffers.outputQueueLength,43.Sink__bad_records.numRecordsOutPerSecond,92.numBytesInLocal,110.numRecordsInPerSecond,11.numRecordsInPerSecond,39.buffers.outputQueueLength,103.buffers.outputQueueLength,131.buffers.outPoolUsage,53.numRecordsIn,62.numBytesOutPerSecond,63.numBytesInLocal,110.numRecordsOut,100.numBytesInLocal,124.numRecordsInPerSecond,56.Sink__bad_records.numRecordsInPerSecond,48.buffers.outPoolUsage,32.Sink__bad_records.numRecordsInPerSecond,140.numBytesOut,80.lastCheckpointSize,93.numRecordsInPerSecond,85.buffers.outPoolUsage,59.Sink__bad_records.numRecordsOutPerSecond,88.numBytesInRemotePerSecond,53.numBytesOutPerSecond,59.numRecordsInPerSecond,15.Sink__bad_records.numRecordsIn,11.Sink__bad_records.latency,132.currentLowWatermark,125.numBytesInLocalPerSecond,78.buffers.outPoolUsage,133.Sink__bad_records.numRecordsIn,68.numBytesInRemote,94.lastCheckpointSize,30.numBytesInLocalPerSecond,107.buffers.inPoolUsage,85.Sink__bad_records.numRecordsInPerSecond,90.numBytesOutPerSecond,99.numRecordsIn,36.Sink__bad_records.numRecordsOutPerSecond,74.numBytesInRemotePerSecond,16.Sink__bad_records.numRecordsOutPerSecond,138.numRecordsIn,6.buffers.inPoolUsage,9.numBytesInRemote,93.currentLowWatermark,138.numRecordsOut,118.Sink__bad_records.numRecordsOut,30.numBytesInLocal,54.numBytesInLocal,120.numRecordsOut,116.buffers.outputQueueLength,90.numBytesInRemotePerSecond,123.Sink__bad_records.numRecordsOut,21.numBytesInLocal,27.numBytesInLocal,39.numRecordsOutPerSecond,20.numRecordsInPerSecond,18.numBytesInLocal,74.currentLowWatermark,8.lastCheckpointSize,114.numRecordsIn,52.Sink__bad_records.numRecordsOutPerSecond,65.numRecordsOutPerSecond,132.buffers.outputQueueLength,90.buffers.outputQueueLength,55.numRecordsIn,42.numBytesOut,118.lastCheckpointSize,53.numRecordsOutPerSecond,57.checkpointAlignmentTime,27.numBytesInLocalPerSecond,44.numBytesInRemote,84.Sink__bad_records.numRecordsInPerSecond,22.numRecordsOut,101.Sink__bad_records.numRecordsOut,121.Sink__bad_records.numRecordsOutPerSecond,15.Sink__bad_records.latency,107.numBytesInLocal,68.buffers.inPoolUsage,138.numBytesInLocalPerSecond,37.numBytesInLocal,137.Sink__bad_records.numRecordsOut,21.Sink__bad_records.numRecordsIn,91.numBytesInRemote,110.numBytesInLocal,27.checkpointAlignmentTime,116.numBytesInLocal,128.numRecordsOut,37.currentLowWatermark,142.Sink__bad_records.numRecordsOut,130.numBytesInLocalPerSecond,51.buffers.outputQueueLength,13.numBytesInLocal,106.Sink__bad_records.numRecordsOutPerSecond,133.numRecordsInPerSecond,124.buffers.outPoolUsage,61.numBytesInRemotePerSecond,128.Sink__bad_records.numRecordsOutPerSecond,72.lastCheckpointSize,117.buffers.outPoolUsage,29.Sink__bad_records.latency,103.buffers.outPoolUsage,99.Sink__bad_records.numRecordsOut,76.buffers.inputQueueLength,77.numRecordsIn,79.buffers.inputQueueLength,23.Sink__bad_records.numRecordsOutPerSecond,45.Sink__bad_records.numRecordsOutPerSecond,14.Sink__bad_records.numRecordsInPerSecond,69.numBytesInRemote,36.numRecordsInPerSecond,70.numBytesInLocalPerSecond,123.numBytesInRemote,20.checkpointAlignmentTime,140.checkpointAlignmentTime,16.Sink__bad_records.numRecordsOut,13.Sink__bad_records.numRecordsInPerSecond,4.currentLowWatermark,59.Sink__bad_records.numRecordsInPerSecond,7.checkpointAlignmentTime,103.lastCheckpointSize,69.Sink__bad_records.latency,84.buffers.outPoolUsage,46.numBytesInLocal,95.buffers.inputQueueLength,64.numBytesOut,53.numBytesOut,40.numBytesInLocal,117.Sink__bad_records.latency,82.numBytesInLocal,64.Sink__bad_records.numRecordsIn,136.numRecordsIn,29.lastCheckpointSize,81.Sink__bad_records.latency,31.numBytesInRemotePerSecond,94.Sink__bad_records.numRecordsOutPerSecond,43.numBytesInLocalPerSecond,32.numRecordsOut,119.numRecordsOutPerSecond,136.lastCheckpointSize,125.Sink__bad_records.numRecordsInPerSecond,69.Sink__bad_records.numRecordsIn,21.Sink__bad_records.numRecordsOut,19.numBytesInRemotePerSecond,47.numBytesOutPerSecond,116.numRecordsOut,116.checkpointAlignmentTime,65.Sink__bad_records.latency,11.numBytesInLocalPerSecond,116.numRecordsIn,94.checkpointAlignmentTime,14.Sink__bad_records.numRecordsOutPerSecond,129.numBytesInRemote,72.Sink__bad_records.numRecordsOut,67.Sink__bad_records.numRecordsOut,78.numBytesInRemotePerSecond,114.lastCheckpointSize,107.numBytesOutPerSecond,111.buffers.outPoolUsage,90.buffers.outPoolUsage,73.Sink__bad_records.numRecordsOut,49.numBytesOutPerSecond,87.numRecordsInPerSecond,70.numRecordsOutPerSecond,11.numBytesInLocal,122.numRecordsOutPerSecond,32.buffers.inputQueueLength,95.currentLowWatermark,82.lastCheckpointSize,57.buffers.outputQueueLength,120.numBytesInRemotePerSecond,93.numBytesInRemotePerSecond,50.lastCheckpointSize,72.buffers.outputQueueLength,81.Sink__bad_records.numRecordsInPerSecond,75.checkpointAlignmentTime,59.numRecordsOutPerSecond,128.Sink__bad_records.numRecordsOut,7.currentLowWatermark,130.numRecordsIn,84.buffers.inputQueueLength,12.Sink__bad_records.numRecordsOut,49.buffers.outputQueueLength,136.numBytesInRemote,23.numBytesInLocal,70.numBytesOutPerSecond,105.numBytesInRemotePerSecond,1.buffers.outputQueueLength,127.buffers.inputQueueLength,71.lastCheckpointSize,108.Sink__bad_records.numRecordsOutPerSecond,87.Sink__bad_records.latency,36.Sink__bad_records.latency,4.numBytesInLocal,84.numBytesInLocalPerSecond,79.Sink__bad_records.latency,116.numBytesInRemote,5.numRecordsIn,110.buffers.inputQueueLength,3.checkpointAlignmentTime,25.Sink__bad_records.numRecordsIn,71.Sink__bad_records.numRecordsIn,121.buffers.inputQueueLength,7.numRecordsInPerSecond,66.checkpointAlignmentTime,112.Sink__bad_records.numRecordsInPerSecond,67.buffers.outPoolUsage,80.buffers.outputQueueLength,95.Sink__bad_records.numRecordsOutPerSecond,56.currentLowWatermark,91.numRecordsIn,125.numRecordsInPerSecond,116.buffers.inputQueueLength,72.numBytesOutPerSecond,84.numBytesInRemote,134.currentLowWatermark,6.numBytesInRemotePerSecond,49.buffers.inputQueueLength,56.numRecordsOutPerSecond,103.numBytesOutPerSecond,127.buffers.outputQueueLength,122.Sink__bad_records.numRecordsIn,139.Sink__bad_records.numRecordsInPerSecond,102.numBytesOut,108.numBytesOut,135.buffers.outputQueueLength,133.currentLowWatermark,93.lastCheckpointSize,128.numBytesInRemote,57.buffers.outPoolUsage,84.numRecordsOut,7.Sink__bad_records.numRecordsOut,74.numBytesOutPerSecond,85.numBytesInRemotePerSecond,76.currentLowWatermark,114.currentLowWatermark,6.lastCheckpointSize,135.checkpointAlignmentTime,17.numRecordsOutPerSecond,110.checkpointAlignmentTime,3.numBytesOut,75.currentLowWatermark,137.numRecordsOut,19.Sink__bad_records.numRecordsIn,92.numBytesInRemote,33.Sink__bad_records.numRecordsInPerSecond,126.buffers.inPoolUsage,121.buffers.outPoolUsage,12.numRecordsInPerSecond,12.numRecordsOut,105.numBytesOutPerSecond,87.buffers.inPoolUsage,9.numBytesOut,69.buffers.outPoolUsage,38.numBytesInLocalPerSecond,129.Sink__bad_records.numRecordsOutPerSecond,13.numRecordsOut,88.numRecordsInPerSecond,7.lastCheckpointSize,56.numBytesInLocal,123.numRecordsIn,73.buffers.inputQueueLength,64.numRecordsOut,83.numRecordsOut,116.numBytesOut,59.numBytesOutPerSecond,90.Sink__bad_records.numRecordsInPerSecond,94.currentLowWatermark,127.numBytesOut,68.checkpointAlignmentTime,104.lastCheckpointSize,94.numRecordsIn,32.Sink__bad_records.numRecordsIn,119.numBytesOut,131.numRecordsInPerSecond,96.numBytesInLocal,111.numBytesInLocalPerSecond,107.currentLowWatermark,109.checkpointAlignmentTime,128.numRecordsOutPerSecond,101.numRecordsIn,60.lastCheckpointSize,44.numBytesInLocal,105.buffers.inputQueueLength,138.Sink__bad_records.numRecordsOut,44.Sink__bad_records.latency,21.buffers.inputQueueLength,42.numRecordsInPerSecond,132.buffers.inputQueueLength,23.numBytesInRemote,135.currentLowWatermark,73.numBytesInLocalPerSecond,5.numBytesInLocalPerSecond,111.numRecordsIn,65.buffers.outPoolUsage,108.numBytesInRemote,143.Sink__bad_records.numRecordsOut,21.Sink__bad_records.numRecordsOutPerSecond,96.currentLowWatermark,54.buffers.inputQueueLength,73.numRecordsOutPerSecond,124.numBytesInRemote,77.numBytesInLocal,143.Sink__bad_records.numRecordsOutPerSecond,67.Sink__bad_records.numRecordsOutPerSecond,24.numBytesInRemotePerSecond,120.numBytesOutPerSecond,84.Sink__bad_records.latency,75.buffers.outPoolUsage,72.numRecordsIn,105.numBytesOut,5.Sink__bad_records.numRecordsInPerSecond,17.Sink__bad_records.numRecordsOut,14.buffers.outputQueueLength,50.numBytesInRemotePerSecond,96.numBytesInRemote,115.buffers.outPoolUsage,143.numBytesOut,57.numRecordsInPerSecond,100.numBytesInLocalPerSecond,80.numBytesOutPerSecond,10.buffers.inputQueueLength,14.numRecordsOutPerSecond,138.buffers.inPoolUsage,5.buffers.inputQueueLength,68.currentLowWatermark,58.Sink__bad_records.numRecordsIn,38.buffers.inputQueueLength,125.numRecordsOutPerSecond,133.numRecordsIn,5.currentLowWatermark,75.Sink__bad_records.numRecordsIn,63.numRecordsOut,42.Sink__bad_records.numRecordsInPerSecond,115.lastCheckpointSize,68.Sink__bad_records.numRecordsOut,122.Sink__bad_records.latency,72.numRecordsInPerSecond,64.numBytesInRemote,42.numBytesInRemotePerSecond,22.Sink__bad_records.numRecordsOut,38.Sink__bad_records.numRecordsOutPerSecond,6.numBytesOut,88.Sink__bad_records.numRecordsOutPerSecond,16.numBytesInRemotePerSecond,113.numBytesOutPerSecond,49.currentLowWatermark,82.buffers.outPoolUsage,1.numBytesOutPerSecond,119.buffers.inPoolUsage,3.buffers.outPoolUsage,114.numBytesInLocal,19.lastCheckpointSize,5.Sink__bad_records.latency,126.lastCheckpointSize,59.checkpointAlignmentTime,126.numBytesInLocal,49.numBytesInLocalPerSecond,114.Sink__bad_records.latency,2.numRecordsOutPerSecond,62.numRecordsIn,108.Sink__bad_records.numRecordsIn,110.buffers.outputQueueLength,106.currentLowWatermark,67.numBytesInRemotePerSecond,9.Sink__bad_records.numRecordsOutPerSecond,104.numRecordsIn,140.Sink__bad_records.latency,68.numBytesInLocalPerSecond,89.buffers.inPoolUsage,36.numBytesInRemote,82.numBytesInRemotePerSecond,93.Sink__bad_records.numRecordsOut,42.numBytesInLocal,135.Sink__bad_records.numRecordsInPerSecond,34.checkpointAlignmentTime,143.buffers.inputQueueLength,41.numRecordsInPerSecond,5.checkpointAlignmentTime,43.numRecordsOut,89.Sink__bad_records.numRecordsOutPerSecond,142.checkpointAlignmentTime,11.numBytesInRemote,66.numBytesOutPerSecond,79.numRecordsOutPerSecond,116.currentLowWatermark,98.Sink__bad_records.numRecordsInPerSecond,76.Sink__bad_records.latency,40.numRecordsIn,125.Sink__bad_records.latency,71.Sink__bad_records.numRecordsInPerSecond,140.buffers.inputQueueLength,65.Sink__bad_records.numRecordsInPerSecond,111.Sink__bad_records.numRecordsIn,96.numBytesInRemotePerSecond,88.Sink__bad_records.numRecordsOut,112.numBytesInLocal,97.Sink__bad_records.numRecordsIn,87.numRecordsIn,53.numBytesInRemotePerSecond,4.buffers.outputQueueLength,82.Sink__bad_records.numRecordsIn,23.Sink__bad_records.numRecordsInPerSecond,73.buffers.outPoolUsage,62.buffers.inputQueueLength,141.Sink__bad_records.numRecordsInPerSecond,65.numBytesOutPerSecond,51.buffers.inputQueueLength,46.buffers.outputQueueLength,36.Sink__bad_records.numRecordsIn,72.buffers.outPoolUsage,0.buffers.outPoolUsage,126.Sink__bad_records.numRecordsIn,3.numBytesInRemotePerSecond,80.numBytesInLocal,11.numRecordsOutPerSecond,28.numRecordsInPerSecond,73.Sink__bad_records.numRecordsOutPerSecond,102.numRecordsInPerSecond,97.buffers.outputQueueLength,70.lastCheckpointSize,64.numBytesOutPerSecond,108.numBytesInRemotePerSecond,51.numBytesInRemote,108.buffers.inputQueueLength,109.numBytesInLocal,127.buffers.inPoolUsage,128.buffers.inPoolUsage,8.numRecordsInPerSecond,58.numRecordsInPerSecond,4.numBytesInRemote,110.numBytesOutPerSecond,94.numRecordsInPerSecond,138.numBytesOut,18.Sink__bad_records.latency,132.numBytesOut,76.numBytesInRemote,77.currentLowWatermark,112.checkpointAlignmentTime,13.buffers.inputQueueLength,102.numRecordsOutPerSecond,105.buffers.outPoolUsage,74.buffers.outPoolUsage,139.Sink__bad_records.latency,71.numRecordsInPerSecond,66.Sink__bad_records.numRecordsOutPerSecond,78.Sink__bad_records.numRecordsOut,132.numRecordsInPerSecond,76.numRecordsOutPerSecond,75.numBytesInLocal,32.Sink__bad_records.numRecordsOut,2.buffers.outPoolUsage,53.numRecordsOut,69.buffers.inPoolUsage,93.numRecordsOutPerSecond,73.checkpointAlignmentTime,38.currentLowWatermark,58.currentLowWatermark,108.buffers.inPoolUsage,39.numBytesInLocal,131.numBytesInRemotePerSecond,109.buffers.outputQueueLength,27.Sink__bad_records.numRecordsOut,88.buffers.inPoolUsage,112.numBytesOutPerSecond,19.buffers.inputQueueLength,124.buffers.outputQueueLength,107.buffers.outPoolUsage,83.buffers.outputQueueLength,57.currentLowWatermark,116.Sink__bad_records.numRecordsInPerSecond,120.buffers.outPoolUsage,25.numBytesInLocal,29.currentLowWatermark,27.buffers.inputQueueLength,83.Sink__bad_records.numRecordsOut,11.checkpointAlignmentTime,121.numBytesOut,105.numRecordsOutPerSecond,80.buffers.outPoolUsage,28.numBytesInRemote,58.numBytesOutPerSecond,1.Sink__bad_records.numRecordsInPerSecond,24.buffers.inputQueueLength,29.checkpointAlignmentTime,113.numBytesOut,107.checkpointAlignmentTime,124.numBytesOut,7.Sink__bad_records.numRecordsIn,137.checkpointAlignmentTime,37.numBytesOut,115.currentLowWatermark,98.buffers.inPoolUsage,92.lastCheckpointSize,98.checkpointAlignmentTime,119.buffers.inputQueueLength,81.buffers.outPoolUsage,42.Sink__bad_records.numRecordsOut,125.currentLowWatermark,46.Sink__bad_records.numRecordsInPerSecond,43.buffers.inputQueueLength,33.Sink__bad_records.latency,6.currentLowWatermark,120.Sink__bad_records.numRecordsOutPerSecond,86.currentLowWatermark,81.lastCheckpointSize,41.checkpointAlignmentTime,138.buffers.outputQueueLength,138.buffers.inputQueueLength,59.buffers.outPoolUsage,4.numRecordsOut,2.Sink__bad_records.numRecordsOut,136.Sink__bad_records.numRecordsOutPerSecond,57.numBytesOutPerSecond,40.buffers.inputQueueLength,65.numRecordsIn,15.Sink__bad_records.numRecordsOutPerSecond,135.buffers.inputQueueLength,23.numRecordsOut,95.numBytesInLocalPerSecond,115.Sink__bad_records.numRecordsIn,116.lastCheckpointSize,67.currentLowWatermark,13.numBytesInRemotePerSecond,47.Sink__bad_records.numRecordsIn,111.numBytesOutPerSecond,66.buffers.outPoolUsage,86.Sink__bad_records.numRecordsIn,127.numRecordsOut,73.Sink__bad_records.latency,106.buffers.outPoolUsage,94.numBytesInLocal,87.numBytesInLocalPerSecond,47.Sink__bad_records.latency,58.numBytesInLocal,94.Sink__bad_records.numRecordsInPerSecond,135.numBytesOut,137.buffers.inPoolUsage,1.buffers.outPoolUsage,26.numBytesOut,31.numBytesInRemote,15.numBytesOut,43.buffers.outputQueueLength,90.numRecordsOutPerSecond,37.Sink__bad_records.numRecordsOut,69.Sink__bad_records.numRecordsInPerSecond,112.Sink__bad_records.numRecordsOut,48.currentLowWatermark,111.Sink__bad_records.latency,6.numBytesInLocal,43.Sink__bad_records.numRecordsIn,16.buffers.inputQueueLength,79.numBytesInLocalPerSecond,138.lastCheckpointSize,104.numBytesOutPerSecond,117.numRecordsOut,110.numBytesOut,105.lastCheckpointSize,27.numBytesInRemotePerSecond,119.Sink__bad_records.numRecordsIn,8.Sink__bad_records.latency,113.buffers.outPoolUsage,126.numRecordsIn,127.lastCheckpointSize,3.Sink__bad_records.numRecordsIn,118.numRecordsInPerSecond,108.numRecordsOutPerSecond,79.buffers.inPoolUsage,107.Sink__bad_records.numRecordsOutPerSecond,33.numRecordsOut,107.Sink__bad_records.numRecordsOut,17.buffers.outputQueueLength,2.numRecordsIn,44.Sink__bad_records.numRecordsOutPerSecond,118.buffers.inPoolUsage,42.numRecordsOutPerSecond,56.numBytesInRemote

Chesnay Schepler

Re: Tame Flink UI?

Hello,

So there are 2 separate issues here:

The response when requesting the list of available metrics is pretty big.
The request for the values of these metrics is also pretty big, and the response even larger.

For now I will modify the WebUI to only ask the value of selected metrics, which will typically be a rather small number. This
should solve issue #2. (Provided no one has the funny idea of selecting all metrics!)

To fix #1 we will have to go with a more compact representation i guess; this however will require a bit more work,+ since the
backend has to detect the valid ranges (i.e the subtasks for which we do in fact have the metric).

Note that #1 should not be such a big problem since the list of metrics is not update regularly as far as i know.
If one doesn't interact with the metrics tab no request (or maybe one at startup) should be sent.

Regards,
Chesnay

On 16.11.2016 20:38, Cliff Resnick wrote:

Ufuk,

The above occurs for me simply by selecting a running job from the job list.

Chesnay,

The 413 error is because of the large request size. Given all the repeated parameter names maybe a more compact representation would work? For example, instead of enumerating all metrics, maybe ask for the range?

On Wed, Nov 16, 2016 at 2:05 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

The WebInterfaces first pulls a list of all available metrics for a specific taskmanager/job/task (which is reasonable since how else would you select them),
and then requests the values for all metrics by supplying the name of every single metric it just received, which is where things get funky.

In this case we have a task with parallelism of around 90 (the number before the metric name is the subtask index).
Now let's only consider IO metrics (numRecordsIn etc.).
We then have 90 * 6 (task IO metrics) + 90 * 4 (operator IO metrics) * X (# of operators in the task) metrics.
In the best of case of a single operator this results in 900 metrics being pulled at once,
which is done every few seconds; i don't know the exact update interval.

We can disable this temporarily in a few ways; the easiest one being to simply never return any metrics in the initial metrics look-up.
See AbstractMetricsHandler#getAvailableMetricsList

Regards,
Chesnay

On 16.11.2016 19:15, Ufuk Celebi wrote:

Hey Cliff,

yes this has been recently merged to the master branch. I think you are right that this is not feasible. I thought that the metrics are pulled in selectively when you select them via the metrics list. It seems to be not the case.

If it is really the case that everything is always requested then we would have to revert this for the time being. Did you select any metrics manually?

– Ufuk

On 16 November 2016 at 18:40:52, Cliff Resnick ([hidden email]) wrote:

We're on 1.2-SNAPSHOT, and some time over the past couple of weeks the UI
seems to have become much more aggressive polling for metrics. I'm seeing
hundreds of 413 errors as the UI continuously tries to GET with a URL over
100k, pretty much overwhelming the SOCKS proxy connection.
Below is an example of a javascript GET that seems to be emitted every few
seconds. Is this intentional? <removed by Chesnay>