(DEPRECATED) Apache Flink User Mailing List archive.

Stream processing into single sink to multiple DB Schemas

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
Tamir Sagi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Stream processing into single sink to multiple DB Schemas

Tamir Sagi
Hey Community

Assuming there are 3 groups,
A, B, C

Each group represents a set of data about employees and salaries.
Group A ( 0-20K $)
Group B (20K$ - 50K$)
Group C ( > 50K$)

Is it possible to process stream data from single source containing information about employees and salaries and split the data into different DB schemas on the same DB? (Single Sink - Single Connection)

I Encountered Side output and dynamic tables

I'm not sure it's the right way.

If there is a better way , enlighten me

Thank you,

Tamir.




Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. 
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Nicolaus Weidner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stream processing into single sink to multiple DB Schemas

Nicolaus Weidner
Hi Tamir,

I assume you want to use the Jdbc connector?
You can use three filters on your input stream to separate it into three separate streams, then add a sink to each of those (see e.g. [1]). Then you can have a different SQL statement for each of the three sinks. If you specify the driver name in JdbcConnectionOptions, that driver will be used to obtain a DB connection (see [2]). So if you use a pooling driver (e.g. [3]), connections should automatically be taken from a shared pool.

Does that help?

Best wishes,
Nico


On Mon, Jun 7, 2021 at 8:23 AM Tamir Sagi <[hidden email]> wrote:
Hey Community

Assuming there are 3 groups,
A, B, C

Each group represents a set of data about employees and salaries.
Group A ( 0-20K $)
Group B (20K$ - 50K$)
Group C ( > 50K$)

Is it possible to process stream data from single source containing information about employees and salaries and split the data into different DB schemas on the same DB? (Single Sink - Single Connection)

I Encountered Side output and dynamic tables

I'm not sure it's the right way.

If there is a better way , enlighten me

Thank you,

Tamir.




Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. 
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Maciej Obuchowski
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stream processing into single sink to multiple DB Schemas

Maciej Obuchowski
Hey,
We had similar problem, but with 1000s of tables. I've created issue [1] and PR with internally used solution [2], but unfortunately, there seems to be no interest in upstreaming this feature.

Thanks,
Maciej


pon., 7 cze 2021 o 17:15 Nicolaus Weidner <[hidden email]> napisał(a):
Hi Tamir,

I assume you want to use the Jdbc connector?
You can use three filters on your input stream to separate it into three separate streams, then add a sink to each of those (see e.g. [1]). Then you can have a different SQL statement for each of the three sinks. If you specify the driver name in JdbcConnectionOptions, that driver will be used to obtain a DB connection (see [2]). So if you use a pooling driver (e.g. [3]), connections should automatically be taken from a shared pool.

Does that help?

Best wishes,
Nico


On Mon, Jun 7, 2021 at 8:23 AM Tamir Sagi <[hidden email]> wrote:
Hey Community

Assuming there are 3 groups,
A, B, C

Each group represents a set of data about employees and salaries.
Group A ( 0-20K $)
Group B (20K$ - 50K$)
Group C ( > 50K$)

Is it possible to process stream data from single source containing information about employees and salaries and split the data into different DB schemas on the same DB? (Single Sink - Single Connection)

I Encountered Side output and dynamic tables

I'm not sure it's the right way.

If there is a better way , enlighten me

Thank you,

Tamir.




Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. 
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Free forum by Nabble Edit this page