Strange Kafka consumer behaviour

[Gyula Fóra]

Hi,

I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint.

It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10.

Is this expected?

Thanks,

Gyula

[Aljoscha Krettek]

I think it's because it's just taking the state for the existing topics and starts reading from that.

A workaround could be to change the uid of the operator to avoid reading the old state.

Aljoscha

> On 21. Feb 2018, at 02:21, Gyula Fóra <[hidden email]> wrote:
>
> Hi,
> I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint.
>
> It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10.
>
> Is this expected?
>
> Thanks,
> Gyula

[Ankit Chaudhary]

Did you changed the kafka group id? 

On Mon, Mar 19, 2018 at 9:54 PM, Aljoscha Krettek <[hidden email]> wrote:

I think it's because it's just taking the state for the existing topics and starts reading from that.

A workaround could be to change the uid of the operator to avoid reading the old state.

Aljoscha

> On 21. Feb 2018, at 02:21, Gyula Fóra <[hidden email]> wrote:
>
> Hi,
> I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint.
>
> It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10.
>
> Is this expected?
>
> Thanks,
> Gyula

[Tzu-Li (Gordon) Tai]

Hi Gyula,

Are you using Flink 1.4.x, and have partition discovery enabled?
If yes, then both the state of previously existing topics, as well as partitions of the newly specified topics will be consumed.

Cheers,
Gordon

On Tue, Mar 20, 2018 at 6:01 AM, Ankit Chaudhary <[hidden email]> wrote:

Did you changed the kafka group id? 

On Mon, Mar 19, 2018 at 9:54 PM, Aljoscha Krettek <[hidden email]> wrote:

I think it's because it's just taking the state for the existing topics and starts reading from that.

A workaround could be to change the uid of the operator to avoid reading the old state.

Aljoscha

> On 21. Feb 2018, at 02:21, Gyula Fóra <[hidden email]> wrote:
>
> Hi,
> I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint.
>
> It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10.
>
> Is this expected?
>
> Thanks,
> Gyula