Async operator with a KeyedStream

[bastien dine]

Hello,

I would like to know if you can use a KeyedStream with the Async operator :

I want to use the async operator to insert some stuff in my database but I want to limit 1 request per element (with key=id) at a time

With a regular keyBy / map, it's working, but it's too slow (i don't have enough ressources to increase my parallelism), 

As far as I have seen, this is not possible

When I write something like Async.orderedWait(myStream.keyBy(myKeyselector)), the keyBy is totally ignored

Have you a solution for this?

Best Regards,

Bastien

[vino yang]

Hi Bastien,

Your analysis of using KeyedStream in Async I/O is correct. It will not figure out the key.

In your scene, the good practice about interacting with DB is async I/O + thread pool[1] + connection Pool.

You can use a connection pool to reuse and limit the mysql connection.

Best,

Vino

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html#implementation-tips

bastien dine <[hidden email]> 于2019年10月31日周四 下午4:36写道：

Hello,

I would like to know if you can use a KeyedStream with the Async operator :

I want to use the async operator to insert some stuff in my database but I want to limit 1 request per element (with key=id) at a time

With a regular keyBy / map, it's working, but it's too slow (i don't have enough ressources to increase my parallelism), 

As far as I have seen, this is not possible

When I write something like Async.orderedWait(myStream.keyBy(myKeyselector)), the keyBy is totally ignored

Have you a solution for this?

Best Regards,

Bastien

------------------

Bastien DINE
Data Architect / Software Engineer / Sysadmin

bastiendine.io