How do I increase number of db connections of the Flink JDBC Connector?

[Li Peng-2]

Hey folks, 

I'm trying to use flink to write high throughput incoming data to a SQL db using the JDBC Connector as described here:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html

However, after enabling this, my data consumption rate slowed down to a crawl. After doing some digging, it seemed like the number of connections to the db was very low, almost 1 connection per worker (there are 9 workers). Which seemed like the culprit if Flink was blocking and waiting for a single db connection to do all this work?

Is there a way to tell the JDBC connector to use more db connections? Or do I need to specifically increase the parallelism of the connector to beyond the default?

And if I need to increase the parallelism, currently my setup is having 9 workers, 1 task slot for each (this cluster is dedicated to one job). That means I have to increase the number of task slots before increasing parallelism, right?

My flink version is 1.10.1 and my jdbc connection is flink-connector-jdbc_2.11:1.11.0. 

Thanks!
Li

[Chesnay Schepler]

Every works uses exactly 1 connection, so in order to increase the
number of connections you must indeed increase the worker parallelism.

On 2/19/2021 6:51 AM, Li Peng wrote:

> Hey folks,
>
> I'm trying to use flink to write high throughput incoming data to a
> SQL db using the JDBC Connector as described here:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html 
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html>
>
>
> However, after enabling this, my data consumption rate slowed down to
> a crawl. After doing some digging, it seemed like the number of
> connections to the db was very low, almost 1 connection per worker
> (there are 9 workers). Which seemed like the culprit if Flink was
> blocking and waiting for a single db connection to do all this work?
>
> Is there a way to tell the JDBC connector to use more db connections?
> Or do I need to specifically increase the parallelism of the connector
> to beyond the default?
>
> And if I need to increase the parallelism, currently my setup is
> having 9 workers, 1 task slot for each (this cluster is dedicated to
> one job). That means I have to increase the number of task slots
> before increasing parallelism, right?
>
> My flink version is 1.10.1 and my jdbc connection
> is flink-connector-jdbc_2.11:1.11.0.
>
> Thanks!
> Li

[Li Peng-2]

Ah got it, thanks!

On Thu, Feb 18, 2021 at 10:53 PM Chesnay Schepler <[hidden email]> wrote:

Every works uses exactly 1 connection, so in order to increase the
number of connections you must indeed increase the worker parallelism.

On 2/19/2021 6:51 AM, Li Peng wrote:
> Hey folks,
>
> I'm trying to use flink to write high throughput incoming data to a
> SQL db using the JDBC Connector as described here:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html>
>
>
> However, after enabling this, my data consumption rate slowed down to
> a crawl. After doing some digging, it seemed like the number of
> connections to the db was very low, almost 1 connection per worker
> (there are 9 workers). Which seemed like the culprit if Flink was
> blocking and waiting for a single db connection to do all this work?
>
> Is there a way to tell the JDBC connector to use more db connections?
> Or do I need to specifically increase the parallelism of the connector
> to beyond the default?
>
> And if I need to increase the parallelism, currently my setup is
> having 9 workers, 1 task slot for each (this cluster is dedicated to
> one job). That means I have to increase the number of task slots
> before increasing parallelism, right?
>
> My flink version is 1.10.1 and my jdbc connection
> is flink-connector-jdbc_2.11:1.11.0.
>
> Thanks!
> Li