Testing iterative data flows

[Ken Krugler]

Hi all,

What’s the recommended way currently to test a streaming data flow that has iterations?

I know that using timeouts in tests (which FLINK-2390 also discusses) isn’t reliable, and it’s hard to know when a job with iterations is really “done” in the context of a test.

Are there any other approaches with current versions of Flink that would be better than an arbitrary timeout?

Thanks,

— Ken

--------------------------

Ken Krugler

[phone]

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[Ufuk Celebi]

Hey Ken! Unfortunately, no. But Paris just posted a proposal to
improve this: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-13-Consistent-Processing-with-Loops-tt14149.html

On Wed, Oct 26, 2016 at 11:10 PM, Ken Krugler
<[hidden email]> wrote:

> Hi all,
>
> What’s the recommended way currently to test a streaming data flow that has
> iterations?
>
> I know that using timeouts in tests (which FLINK-2390 also discusses) isn’t
> reliable, and it’s hard to know when a job with iterations is really “done”
> in the context of a test.
>
> Are there any other approaches with current versions of Flink that would be
> better than an arbitrary timeout?
>
> Thanks,
>
> — Ken
>
> --------------------------
> Ken Krugler
> [phone]
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>