flink kafka question

[Jerry Peng]

Hello,

I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

Best,

Jerry

[Ufuk Celebi]

> On 12 Oct 2015, at 22:47, Jerry Peng <[hidden email]> wrote:
>
> Hello,
>
> I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

I guess you want to do this in the consumer in order to have less delay from when you read it and the timestamp. I am not familiar with the consumer code, but you can try to this in a map after the source. This should be chained to the source and the delay should not be too large from the source to the map function.

[rmetzger0]

I would also suggest to create a mapper after the source. Make sure the mapper is chained to the kafka source, then, you'll not really see a big delay in the timestamp written to redis.

Just out of curiosity, why do you need to write a timestamp to redis for each record from Kafka?

On Wed, Oct 14, 2015 at 11:40 AM, Ufuk Celebi <[hidden email]> wrote:

> On 12 Oct 2015, at 22:47, Jerry Peng <[hidden email]> wrote:
>
> Hello,
>
> I am trying to do some benchmark testing with flink streaming.  When flink reads a message in from Kafka, I want to write a timestamp to redis.  How can I modify the existing kafka consumer code to do this?  What would be easiest way to do something like this?  Thanks for your help!

I guess you want to do this in the consumer in order to have less delay from when you read it and the timestamp. I am not familiar with the consumer code, but you can try to this in a map after the source. This should be chained to the source and the delay should not be too large from the source to the map function.