Re: Flink - Once and once processing

Posted by snntr on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-Once-and-once-processing-tp8206p8211.html

Hi Mans,

depending on the number of operations and the particular database, you
might be able to use transactions.

Maybe you can also find a data model, which is more resilient to these
kind of failures.

Cheers,

Konstantin

On 29.07.2016 19:26, M Singh wrote:

> Hi:
>
> I have a use case where we need to update a counter in a db and for this
> need to guarantee once only processing.  If we have some entries in a
> batch and it partially updates the counters and then fails, if Flink
> retries the processing for that batch, some of the counters will be
> updated twice (the ones which succeeded in the first batch).
>
> I think in order to guarantee once only processing, I will have to set
> the buffer size to zero (ie, send one item at a time).
>
> Is there any alternative configuration or suggestion on how I can
> achieve once only updates using a batch mode with partial failures ?
>
> Thanks
>
> Mans
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082