Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

[Arvid Heise-3]

I was wondering if you could actually really use AsyncWaitOperator in the following way.

- Use a rather big timeout (so if callbacks usually take 1s, use 10).

- Use UNORDERED mode.

- Use a rather big queue size that would not cause any backpressure (you could just experiment with different settings).

Then, you'd probably get to the operator that you would need to implement manually anyways.

- Requests come in a specific order, that order is retained when calling the external library.

- Results are immediately returned (depending on your watermark settings) resulting in no additional latency (because of UNORDERED).

- The big timeouts guarantee that you will not dismiss a certain input too quickly, if the callback takes longer than usual. It will clean up all elements from state that have no callbacks after the given time though.

- The big queue size will avoid backpressure resulting from many pending requests without response. Let's say you have 100 requests per second and a timeout of 10s, that means a queue size of 1000 would allow all incoming requests to be processed almost instantly (ignored the actual callbacks that decrease the needed queue size as you said it to be a rather rare event)

On Thu, Apr 9, 2020 at 11:09 AM Salva Alcántara <[hidden email]> wrote:

Perfectly understood, thanks a lot for your reply/patience . I will take a
look at AsyncWaitOperator and adapt from there if I really need that.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--

[Salva Alcántara]

Sounds like a plan Arvid! Taking note of it, this is gold!



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/