Re: data enrichment with SQL use case
Posted by
Ken Krugler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/data-enrichment-with-SQL-use-case-tp19520p19523.html
If the SQL data is all (or mostly all) needed to join against the data from Kafka, then I might try a regular join.
Otherwise it sounds like you want to use an AsyncFunction to do ad hoc queries (in parallel) against your SQL DB.
— Ken
Hi,
I have a case of meta data enrichment and im wondering if my approach is the correct way .
- input stream from kafka.
- MD in msSQL .
- map to new pojo
I need to extract a key from the kafka stream and use it to select some values from the sql table .
SO i thought to use the table SQL api in order to select the table MD
then convert the kafka stream to table and join the data by the stream key .
At the end i need to map the joined data to a new POJO and send it to elesticserch .
Any suggestions or different ways to solve this use case ?
thanks,
Miki
--------------------------
Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr