Re: How do I increase number of db connections of the Flink JDBC Connector?

Posted by Li Peng-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-do-I-increase-number-of-db-connections-of-the-Flink-JDBC-Connector-tp41562p41581.html

Ah got it, thanks!

On Thu, Feb 18, 2021 at 10:53 PM Chesnay Schepler <[hidden email]> wrote:

Every works uses exactly 1 connection, so in order to increase the
number of connections you must indeed increase the worker parallelism.

On 2/19/2021 6:51 AM, Li Peng wrote:
> Hey folks,
>
> I'm trying to use flink to write high throughput incoming data to a
> SQL db using the JDBC Connector as described here:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html>
>
>
> However, after enabling this, my data consumption rate slowed down to
> a crawl. After doing some digging, it seemed like the number of
> connections to the db was very low, almost 1 connection per worker
> (there are 9 workers). Which seemed like the culprit if Flink was
> blocking and waiting for a single db connection to do all this work?
>
> Is there a way to tell the JDBC connector to use more db connections?
> Or do I need to specifically increase the parallelism of the connector
> to beyond the default?
>
> And if I need to increase the parallelism, currently my setup is
> having 9 workers, 1 task slot for each (this cluster is dedicated to
> one job). That means I have to increase the number of task slots
> before increasing parallelism, right?
>
> My flink version is 1.10.1 and my jdbc connection
> is flink-connector-jdbc_2.11:1.11.0.
>
> Thanks!
> Li