Hi,
I am trying to simulate the "Split Distinct Aggregation" [1] with the
data from Taxi Ride. I am using the following query:
SELECT dayOfTheYear, COUNT(DISTINCT driverId) FROM TaxiRide GROUP BY
dayOfTheYear
and I am analyzing the different methods for optimizing. So I started
using (1) no optimization, then the (2) "table.exec.mini-batch.size" =
TRUE with "table.optimizer.agg-phase-strategy" = "ONE_PHASE", then I
changed to (3) "table.optimizer.agg-phase-strategy" = "TWO_PHASE", and
finally I use the (4) "table.optimizer.distinct-agg.split.enabled" =
TRUE.
What does the sentence "COUNT DISTINCT is not good at reducing records
if the value of distinct key (i.e. user_id) is sparse." mean on the
website? In my case the distinct key is the driverId. So, should I
change the data source to have a lot of null values on the driverId
column?
I am asking because I tested this query with optimizations (1) and (2)
I got ~20K r/s on each operator (parallelism of 8) when I set the
workload to ~200K r/s. This is almost the total workload. Then I
changed only the optimization to (3, TWO_PHASE) and maximum throughput
reaches only 4K.
I think that the problem is in my data that the query with distinct is
consuming. So, how should I prepare the data to see the optimization
of split distinct take effect?
Thanks,
Felipe
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/tuning/streaming_aggregation_optimization.html#split-distinct-aggregation--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez
--
https://felipeogutierrez.blogspot.com
Locked
