async and checkpointing

[anurag]

Hi ,

Thanks in advance for your help. I am trying to write a flink function which reads from kafka using kafka-flinkconsumer and sends messages to an indexer. I am not clear on how async and checkpointing will work in this case. My flow is like this:

a) Messages are ingested into kafka.

b)The messages are read by kafka-flink consumer from kafka

c)My code is implementing RichAsyncFunction and sends messages to external indexer, I am using threads to send multiple requests to the indexer in parallel.

I am also using unorderedwait strategy for async. I want to know what will happen in following scenarios:

Assuming kafka has 100 messages and say I have 10 threads(numbered T1...T10). Also assume that threads T1 is sending messages 1..10, T2 is sending messages 11..20 asynchronously to the indexer and so on  and say I get reply 200 from indexer for last 8 threads T3-T10 but servers T1,T2 dont get any reply or get non-200 reply. in this case how will the checkpointing pointer move.

Thanks,

Anurag