Sample project on real time ingestion of more than 1Billion events at a time

[Deepak Sharma]

Hi All

Can anyone point me to sample code that can scale upto real time ingestion of events in millions or billion ?

Also i need reference to which analytic engine to store the transformed events for real time querying .

Thanks a lot in advance.

--Deepak

[Timo Walther]

Hi Deepak,

actually, every Flink example program can scale up to millions of events
and more. The Flink APIs are designed to abstract the business logic
from the parallelism. You just need to implement the interfaces that
Flink provides.

If you are interesting in some example program, I can recommend this
blog post [1]. A typical storage after processing, can be Kafka,
EleasticSearch or you skip the additional system and use Flink's
queryable state feature [2].

Hope that helps.

Regards,
Timo

[1] https://data-artisans.com/blog/extending-the-yahoo-streaming-benchmark
[2] https://data-artisans.com/blog/queryable-state-use-case-demo



Am 10/23/17 um 4:42 PM schrieb Deepak Sharma:
> Hi All
> Can anyone point me to sample code that can scale upto real time
> ingestion of events in millions or billion ?
> Also i need reference to which analytic engine to store the
> transformed events for real time querying .
>
> Thanks a lot in advance.
>
> --Deepak