Manually controlling time for integration test

[Maxim Parkachov]

Hi,

I'm evaluating Flink as alternative to Spark streaming for test project reading from Kafka and saving to Cassandra. Everything works, but I'm struggling with integration tests. I could not figure out how to manually move time in Flink. Basically, I write message in Kafka with event time in the future and would like to move clock to force Flink to process this event, like it's possible with manual clock in Spark streaming. Currently, I'm waiting several seconds before I'm checking result in Cassandra, which doesn't seem very efficient to me.

My real use case is a bit more complex, so, I could not use event time in the past, because I'm trying to write tests with multiple messages with different time and test interaction with tumbling windows.

I would appreciate any tip how this could be achieved, link to source code or some other approach to integration tests.

Thanks in advance,

Maxim. 

[Fabian Hueske-2]

Hi Maxim,

you could inject an AssignerWithPunctuatedWatermarks into your plan which emits a watermark for every record it sees.

That way you can increment the logical time for every record.

Best, Fabian

2017-08-04 16:27 GMT+02:00 Maksym Parkachov <[hidden email]>:

Hi,

I'm evaluating Flink as alternative to Spark streaming for test project reading from Kafka and saving to Cassandra. Everything works, but I'm struggling with integration tests. I could not figure out how to manually move time in Flink. Basically, I write message in Kafka with event time in the future and would like to move clock to force Flink to process this event, like it's possible with manual clock in Spark streaming. Currently, I'm waiting several seconds before I'm checking result in Cassandra, which doesn't seem very efficient to me.

My real use case is a bit more complex, so, I could not use event time in the past, because I'm trying to write tests with multiple messages with different time and test interaction with tumbling windows.

I would appreciate any tip how this could be achieved, link to source code or some other approach to integration tests.

Thanks in advance,

Maxim.