How to start with the first Kafka Message

[Dominique Rondé]

Hi folks,

i have a kafka topic with messages from the last 7 days. Now i have a
new flink streaming process and like to consume the messages from the
beginning. If I just bring up the topology, the consumer starts from
this moment and not from beginning.

THX

Dominique

[Till Rohrmann]

Hi Dominique,

have you tried setting the Kafka property props.put("auto.offset.reset", "smallest");?

Cheers,
Till

​

On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé <[hidden email]> wrote:

Hi folks,

i have a kafka topic with messages from the last 7 days. Now i have a new flink streaming process and like to consume the messages from the beginning. If I just bring up the topology, the consumer starts from this moment and not from beginning.

THX

Dominique

[Balaji Rajagopalan]

If it is a one time thing, you could run the kafka consumer script which has the --from-beginning option, and redirect the output as a socketstream and consume in flink. 

On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann <[hidden email]> wrote:

Hi Dominique,

have you tried setting the Kafka property props.put("auto.offset.reset", "smallest");?

Cheers,
Till

​

On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé <[hidden email]> wrote:

Hi folks,

i have a kafka topic with messages from the last 7 days. Now i have a new flink streaming process and like to consume the messages from the beginning. If I just bring up the topology, the consumer starts from this moment and not from beginning.

THX

Dominique

[Dominique Rondé]

I just place the parameterTool.getProperties() in the FlinkKafkaConsumer082 constructor. Now I am able to decide with --auto.offset.reset smallest at the commandline each time i start up a route.

Thanks for your hints!

Dominique

Am 18.03.2016 um 03:02 schrieb Balaji Rajagopalan:

If it is a one time thing, you could run the kafka consumer script which has the --from-beginning option, and redirect the output as a socketstream and consume in flink. 

On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann <[hidden email]> wrote:

Hi Dominique,

have you tried setting the Kafka property props.put("auto.offset.reset", "smallest");?

Cheers,
Till

​

On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé <[hidden email]> wrote:

Hi folks,

i have a kafka topic with messages from the last 7 days. Now i have a new flink streaming process and like to consume the messages from the beginning. If I just bring up the topology, the consumer starts from this moment and not from beginning.

THX

Dominique