how to simulate the scene "back pressure" in flink?Thanks~!
Locked
how to simulate the scene "back pressure" in flink?Thanks~!
 
|
Re: how to simulate the scene "back pressure" in flink?Thanks~!
|
|
The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example. If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test. On Tue, Oct 6, 2020 at 4:13 PM 大森林 <[hidden email]> wrote:
-- Arvid Heise | Senior Java Developer Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbHRegistered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng |
回复： how to simulate the scene "back pressure" in flink?Thanks~!
|
|
Thanks for your repies, could you tell me where to set RestOption.POPT?in configuration what's the value should I set for RestOption.PORT? Thanks. ------------------ 原始邮件 ------------------ 发件人: "Arvid Heise" <[hidden email]>; 发送时间: 2020年10月9日(星期五) 下午3:00 收件人: "大森林"<[hidden email]>; 主题: Re: how to simulate the scene "back pressure" in flink?Thanks~! The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example. If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test. On Tue, Oct 6, 2020 at 4:13 PM 大森林 <[hidden email]> wrote:
-- Arvid Heise | Senior Java Developer Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbHRegistered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng |
Re: how to simulate the scene "back pressure" in flink?Thanks~!
|
|
You can add conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue()); at any place before creating the environment [1]. Default value is 8081, so you can access web ui through http://localhost:8081, but you can really choose any other free port. On Fri, Oct 9, 2020 at 9:24 AM 大森林 <[hidden email]> wrote:
-- Arvid Heise | Senior Java Developer Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbHRegistered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng |
Re: how to simulate the scene "back pressure" in flink?Thanks~!
|
|
The Flink Operations Playground includes an optional backpressure simulation you can experiment with. It is described at the end of [1]. On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <[hidden email]> wrote:
|
回复： how to simulate the scene "back pressure" in flink?Thanks~!
|
|
Thanks for both of your help... but... I can not understand both: ------------------------------------------------------------------------------------------------ Dear David Anderson: Is the whole command like this? flink run --backpressure -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar------------------------------------------------------------------------------------------------ Dear Arvid Heise: For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue()); will this settings work to sleep when the output stream is generating? ------------------------------------------------------------------------------------------------ apologise for my poor basic knowledge of flink~ Thanks for both of your help~ ------------------ 原始邮件 ------------------ 发件人: "David Anderson" <[hidden email]>; 发送时间: 2020年10月9日(星期五) 晚上8:23 收件人: "Arvid Heise"<[hidden email]>; 主题: Re: how to simulate the scene "back pressure" in flink?Thanks~! The Flink Operations Playground includes an optional backpressure simulation you can experiment with. It is described at the end of [1]. On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <[hidden email]> wrote:
|
Re: how to simulate the scene "back pressure" in flink?Thanks~!
|
|
The ClickCountJob used in the operations playground accepts an application parameter, like this: flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092 --checkpointing --event-time --backpressure To try this, you would modify the docker-compose.yaml file in [1]. If you want to see how it is implemented, see [2]. You can not use this --backpressure option with any other application. On Sat, Oct 10, 2020 at 7:26 AM 大森林 <[hidden email]> wrote:
|
回复： how to simulate the scene "back pressure" in flink?Thanks~!
|
|
Could I use your command with no docker? ------------------ 原始邮件 ------------------ 发件人: "David Anderson" <[hidden email]>; 发送时间: 2020年10月10日(星期六) 晚上10:30 收件人: "大森林"<[hidden email]>; 主题: Re: how to simulate the scene "back pressure" in flink?Thanks~! The ClickCountJob used in the operations playground accepts an application parameter, like this: flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092 --checkpointing --event-time --backpressure To try this, you would modify the docker-compose.yaml file in [1]. If you want to see how it is implemented, see [2]. You can not use this --backpressure option with any other application. On Sat, Oct 10, 2020 at 7:26 AM 大森林 <[hidden email]> wrote:
|
Re: how to simulate the scene "back pressure" in flink?Thanks~!
|
|
Could I use your command with no docker? Hypothetically, yes, but it's a somewhat impractical idea. The ClickCountJob needs Flink and Kafka, and there is another java application (the clickevent-generator) that writes into Kafka the data that is being processed. On Sat, Oct 10, 2020 at 5:32 PM 大森林 <[hidden email]> wrote:
|
| Free forum by Nabble | Edit this page |