Anomalous spikes in aggregations of keyed data

[Kegel, Mark]

We have a high volume (600-700 shards) kinesis data stream that we are doing a simple keying and aggregation on. The logic is very simple: kinesis source, key by fields (A,B,C), window (1-minute, tumbling), aggregate by summing over integer field R, connect to sink.

 

We are seeing some anomalous spikes in our aggregations. From one minute to the next, the sum total for one particular key may increase 25x or more and then drop back down to a normal level, yet sums for other keys in the same window remain roughly the same, which we expect.

 

We don’t see this too often. Maybe 1-5 data points (key + timestamp) in an hour’s worth of 1-minute windowed data will have these spikes. The data has fairly low cardinality. There are only roughly two hundred distinct keys. 

 

We inspected the raw kinesis stream and found no duplicates. It isn’t clear how these spikes could happen or what we might do to work around the issue since the code is as idiomatic as possible.

 

We are running the job as part of Kinesis Data Analytics, which is using Flink version 1.8. To connect to Kinesis we are using the amazon-kinesis-connection-flink library (v1.0.4) library and the EFO consumer mode.

 

[Arvid Heise-3]

Hi Mark,

could you double check if these spikes co-occur with checkpointing? If there is an alignment, certain channels are blocked from taking in data. If all keys are more or less contained in a shard with less data, it would why only these keys are affected.

On Mon, Nov 30, 2020 at 9:27 PM Kegel, Mark <[hidden email]> wrote:

We have a high volume (600-700 shards) kinesis data stream that we are doing a simple keying and aggregation on. The logic is very simple: kinesis source, key by fields (A,B,C), window (1-minute, tumbling), aggregate by summing over integer field R, connect to sink.

 

We are seeing some anomalous spikes in our aggregations. From one minute to the next, the sum total for one particular key may increase 25x or more and then drop back down to a normal level, yet sums for other keys in the same window remain roughly the same, which we expect.

 

We don’t see this too often. Maybe 1-5 data points (key + timestamp) in an hour’s worth of 1-minute windowed data will have these spikes. The data has fairly low cardinality. There are only roughly two hundred distinct keys. 

 

We inspected the raw kinesis stream and found no duplicates. It isn’t clear how these spikes could happen or what we might do to work around the issue since the code is as idiomatic as possible.

 

We are running the job as part of Kinesis Data Analytics, which is using Flink version 1.8. To connect to Kinesis we are using the amazon-kinesis-connection-flink library (v1.0.4) library and the EFO consumer mode.

 

--

[Kegel, Mark]

At the moment we checkpoint every minute. I can turn this frequency down but I’m not sure that will fix/hide the issue.

 

Mark

 

From: Arvid Heise <[hidden email]>
Date: Monday, November 30, 2020 at 2:33 PM
To: Kegel, Mark <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: Anomalous spikes in aggregations of keyed data

Hi Mark,

 

could you double check if these spikes co-occur with checkpointing? If there is an alignment, certain channels are blocked from taking in data. If all keys are more or less contained in a shard with less data, it would why only these keys are affected.

 

On Mon, Nov 30, 2020 at 9:27 PM Kegel, Mark <[hidden email]> wrote:

We have a high volume (600-700 shards) kinesis data stream that we are doing a simple keying and aggregation on. The logic is very simple: kinesis source, key by fields (A,B,C), window (1-minute, tumbling), aggregate by summing over integer field R, connect to sink.

 

We are seeing some anomalous spikes in our aggregations. From one minute to the next, the sum total for one particular key may increase 25x or more and then drop back down to a normal level, yet sums for other keys in the same window remain roughly the same, which we expect.

 

We don’t see this too often. Maybe 1-5 data points (key + timestamp) in an hour’s worth of 1-minute windowed data will have these spikes. The data has fairly low cardinality. There are only roughly two hundred distinct keys. 

 

We inspected the raw kinesis stream and found no duplicates. It isn’t clear how these spikes could happen or what we might do to work around the issue since the code is as idiomatic as possible.

 

We are running the job as part of Kinesis Data Analytics, which is using Flink version 1.8. To connect to Kinesis we are using the amazon-kinesis-connection-flink library (v1.0.4) library and the EFO consumer mode.

 

--

Arvid Heise | Senior Java Developer

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng