Re: Stream processing into single sink to multiple DB Schemas

Posted by Nicolaus Weidner on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Stream-processing-into-single-sink-to-multiple-DB-Schemas-tp44244p44259.html

Hi Tamir,

I assume you want to use the Jdbc connector?

You can use three filters on your input stream to separate it into three separate streams, then add a sink to each of those (see e.g. [1]). Then you can have a different SQL statement for each of the three sinks. If you specify the driver name in JdbcConnectionOptions, that driver will be used to obtain a DB connection (see [2]). So if you use a pooling driver (e.g. [3]), connections should automatically be taken from a shared pool.

Does that help?

Best wishes,

Nico

[1] https://stackoverflow.com/questions/53588554/apache-flink-using-filter-or-split-to-split-a-stream

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/jdbc/#jdbc-connection-parameters

[3] https://commons.apache.org/proper/commons-dbcp/

On Mon, Jun 7, 2021 at 8:23 AM Tamir Sagi <[hidden email]> wrote:

Hey Community

Assuming there are 3 groups,

A, B, C

Each group represents a set of data about employees and salaries.

Group A ( 0-20K $)

Group B (20K$ - 50K$)

Group C ( > 50K$)

Is it possible to process stream data from single source containing information about employees and salaries and split the data into different DB schemas on the same DB? (Single Sink - Single Connection)

I Encountered Side output and dynamic tables

https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/side_output/

https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/concepts/dynamic_tables/

I'm not sure it's the right way.

If there is a better way , enlighten me

Thank you,

Tamir.

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. 
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.