Blocking in an Async function

[Olivier Nouguier]

Hi,

  We have a rather classical situation where we use an AsyncRich function to read from Cassandra (C*) and enrich some domain elements.

It works great, but some time, C* is not reachable, how could we handle this in async function ?

 0/ Let it crash && restart.

 1/ Thread.sleep() and retry seem a terrible idea to me.

 2/ handle timeout / error and schedule a retry: is it possible in asyncInvoke?

 3/ Queueing inflight (incomplete) élement and retry later in a sync/blocking function.

Thanks.

--

[Arian Rohani]

Den tors 29 apr. 2021 kl 11:52 skrev Olivier Nouguier <[hidden email]>:

Hi,

  We have a rather classical situation where we use an AsyncRich function to read from Cassandra (C*) and enrich some domain elements.

It works great, but some time, C* is not reachable, how could we handle this in async function ?

 0/ Let it crash && restart.

 1/ Thread.sleep() and retry seem a terrible idea to me.

 2/ handle timeout / error and schedule a retry: is it possible in asyncInvoke?

 3/ Queueing inflight (incomplete) élement and retry later in a sync/blocking function.

Thanks.

--

Olivier Nouguier

Senior Software Engineer

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

[Arvid Heise-4]

Hi Oliver,

if you are truly running async in asyncIO (remember you need an external thread pool provided by an async lib or by yourself), then option 1) sounds best to me. Could you elaborate why you'd think it's terrible? The only downside is that you'd need to disable timeout in asyncIO and handle timeouts yourself.

There is no support for option 2). If you are using ordered asyncIO, I'm also not sure if it makes much sense or what the benefit is over option 1.

Option 3 is certainly possible but I'm not seeing the benefit immediately - if C is down, you cannot progress anyways. Same is true for Option 0.

On Thu, Apr 29, 2021 at 11:54 AM Arian Rohani <[hidden email]> wrote:

Is this your own implementation of the AsyncRich function?

There is already an API available for Flink for async I/O for external data access that solves some of the challenges that you seem to be encountering.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html

Best,

Arian Rohani

Den tors 29 apr. 2021 kl 11:52 skrev Olivier Nouguier <[hidden email]>:

Hi,

  We have a rather classical situation where we use an AsyncRich function to read from Cassandra (C*) and enrich some domain elements.

It works great, but some time, C* is not reachable, how could we handle this in async function ?

 0/ Let it crash && restart.

 1/ Thread.sleep() and retry seem a terrible idea to me.

 2/ handle timeout / error and schedule a retry: is it possible in asyncInvoke?

 3/ Queueing inflight (incomplete) élement and retry later in a sync/blocking function.

Thanks.

--

Olivier Nouguier

Senior Software Engineer

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.