Changing number of partitions for a topic

[Suraj Puvvada]

Hello,

I have a flink job that reads from a source topic that currently has 4 partitions and I need to increase the partition count to 8. 

Do you need to restart the job for that to take effect ?

How does it work in case there is persistent state (like a window operator) involved ?

Any design documents on how partition mapping works would be very helpful. 

Thanks

Suraj

[Benchao Li]

Hi Suraj,

There is a config option[1] to enable partition discovery, which is disabled by default. 

The community discussed to enable it by default[2], but only aims to the new Source API.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#partition-discovery

[2] https://issues.apache.org/jira/browse/FLINK-15703 

Suraj Puvvada <[hidden email]> 于2020年4月21日周二 上午6:01写道：

Hello,

I have a flink job that reads from a source topic that currently has 4 partitions and I need to increase the partition count to 8. 

Do you need to restart the job for that to take effect ?

How does it work in case there is persistent state (like a window operator) involved ?

Any design documents on how partition mapping works would be very helpful. 

Thanks

Suraj

--

[Suraj Puvvada]

Thanks, I'll check it out. 

On Mon, Apr 20, 2020 at 6:23 PM Benchao Li <[hidden email]> wrote:

Hi Suraj,

There is a config option[1] to enable partition discovery, which is disabled by default. 

The community discussed to enable it by default[2], but only aims to the new Source API.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#partition-discovery

[2] https://issues.apache.org/jira/browse/FLINK-15703 

Suraj Puvvada <[hidden email]> 于2020年4月21日周二 上午6:01写道：

Hello,

I have a flink job that reads from a source topic that currently has 4 partitions and I need to increase the partition count to 8. 

Do you need to restart the job for that to take effect ?

How does it work in case there is persistent state (like a window operator) involved ?

Any design documents on how partition mapping works would be very helpful. 

Thanks

Suraj

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]