|
|
1. How can I create a kafka table that can use headers and map them to columns? Currently, I am using KafkaDeserilizationSchema to create a DataStream, and then I convert that DataStream into a Table. I would like to use a more direct approach.
2. What is the recommended way to enrich a kafka table or data-stream with data-from postgres?
a) kafka table and JDBC temporal dimension table with temporal join and lookup cache setup
b) data-stream with async io which connects via JDBC. (note that asycio does not support Keyed State cache)
c) data-stream rich function or process function that uses Keyed State.
3. When using a kafka told and JDBC temporal dimension table how do I prevent N + 1 queries per join row?
When I issued a query such as this:
SELECT k.name, t1.id, t2.metadata, SUM(k.cost)
FROM kafka_table AS k
JOIN jdbc_table_one AS t1 ON k.t1_id = t1.ID
LEFT JOIN jdbc_table_two FOR SYSTEM_TIME AS OF k.proc_time AS t2 ON t1.t2_id = t2.id AND t2.name = k.name
GROUP BY TUMBLE (k.proc_time, INTERVAL '3' MINUTE), k.name, t1.id, t2.metadata
My PostgreSQL sql logs show that jdbc_table_two has a query per each distinct t2.name.
In a real production system, that would be 200,000 queries!
4. When using a JDBC temporal dimension table does Flink retrieve the from the database asynchronously , or is it possible for Flink to multiple join rows at time with a IN (subquery) syntax?
|
Locked
