flink kafka question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

flink kafka question

Jerry Peng
Hello,

I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

Best,

Jerry
Reply | Threaded
Open this post in threaded view
|

Re: flink kafka question

Ufuk Celebi

> On 12 Oct 2015, at 22:47, Jerry Peng <[hidden email]> wrote:
>
> Hello,
>
> I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

I guess you want to do this in the consumer in order to have less delay from when you read it and the timestamp. I am not familiar with the consumer code, but you can try to this in a map after the source. This should be chained to the source and the delay should not be too large from the source to the map function.

Reply | Threaded
Open this post in threaded view
|

Re: flink kafka question

rmetzger0
I would also suggest to create a mapper after the source. Make sure the mapper is chained to the kafka source, then, you'll not really see a big delay in the timestamp written to redis.

Just out of curiosity, why do you need to write a timestamp to redis for each record from Kafka?

On Wed, Oct 14, 2015 at 11:40 AM, Ufuk Celebi <[hidden email]> wrote:

> On 12 Oct 2015, at 22:47, Jerry Peng <[hidden email]> wrote:
>
> Hello,
>
> I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

I guess you want to do this in the consumer in order to have less delay from when you read it and the timestamp. I am not familiar with the consumer code, but you can try to this in a map after the source. This should be chained to the source and the delay should not be too large from the source to the map function.