SQL Performance

[Nicholas Walton]

I’m streaming records down to an Embedded Derby database, at a rate of around 200 records per second. I’m certain Derby can sustain a higher throughput than that, if I could buffer the records but it seems that I’m writing each record as soon as it arrives and as a single transaction which is inefficient.

Is there a way to to improve my throughput rate?

Nick Walton

[vino yang]

Hi Nick,

Can you provide more details? Are you using JDBCOutputFormat? If yes, can `JDBCOutputFormatBuilder#setBatchInterval` help you?

Best,

Vino

Nicholas Walton <[hidden email]> 于2019年11月26日周二 下午9:20写道：

I’m streaming records down to an Embedded Derby database, at a rate of around 200 records per second. I’m certain Derby can sustain a higher throughput than that, if I could buffer the records but it seems that I’m writing each record as soon as it arrives and as a single transaction which is inefficient.

Is there a way to to improve my throughput rate?

Nick Walton

[Jingsong Li]

Hi Nicholas,

You can take a look to JDBCSinkFunction and JDBCUpsertSinkFunction.

They can set something like flush batch max size just like vino mentioned.

JDBCUpsertOutputFormat also has a async thread to flush batch to JDBC

 to avoid too high delay. In this way, the throughput can be improved while

 the delay is low.

Best,

Jingsong Lee

On Wed, Nov 27, 2019 at 10:16 AM vino yang <[hidden email]> wrote:

Hi Nick,

Can you provide more details? Are you using JDBCOutputFormat? If yes, can `JDBCOutputFormatBuilder#setBatchInterval` help you?

Best,

Vino

Nicholas Walton <[hidden email]> 于2019年11月26日周二 下午9:20写道：

I’m streaming records down to an Embedded Derby database, at a rate of around 200 records per second. I’m certain Derby can sustain a higher throughput than that, if I could buffer the records but it seems that I’m writing each record as soon as it arrives and as a single transaction which is inefficient.

Is there a way to to improve my throughput rate?

Nick Walton

--