Flink Jdbc streaming source support in 1.7.1 or in future?

[Manjusha Vuyyuru]

Hello,

Do flink 1.7.1 supports connection to relational database(mysql)?

I want to use mysql as my streaming source to read some configuration..

Thanks,

Manju

[Zhenghua Gao]

Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
read data from a database.
u can have a try.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Manjusha Vuyyuru]

But 'JDBCInputFormat' will exit once its done reading all data.I need something like which keeps polling to mysql and fetch if there are any updates or changes.

Thanks,

manju

On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <[hidden email]> wrote:

Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
read data from a database.
u can have a try.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Fabian Hueske-2]

I think this is very hard to build in a generic way.

The common approach here would be to get access to the changelog stream of the table, writing it to a message queue / event log (like Kafka, Pulsar, Kinesis, ...) and ingesting the changes from the event log into a Flink application.

You can of course build a custom source that returns all added rows if you have some meta data in the table, e.g., timestamps indicating when a row was added.

However, this can easily become very complex, for example if you need to handle deletes and updates.

Best, Fabian

Am Mi., 23. Jan. 2019 um 08:14 Uhr schrieb Manjusha Vuyyuru <[hidden email]>:

But 'JDBCInputFormat' will exit once its done reading all data.I need something like which keeps polling to mysql and fetch if there are any updates or changes.

Thanks,

manju

On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <[hidden email]> wrote:

Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
read data from a database.
u can have a try.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Puneet Kinra-2]

Then common way is to read in the cdc .writing generic operator wont be easy .

On Wed, Jan 23, 2019 at 12:45 PM Manjusha Vuyyuru <[hidden email]> wrote:

But 'JDBCInputFormat' will exit once its done reading all data.I need something like which keeps polling to mysql and fetch if there are any updates or changes.

Thanks,

manju

On Wed, Jan 23, 2019 at 7:10 AM Zhenghua Gao <[hidden email]> wrote:

Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to
read data from a database.
u can have a try.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--