Live configuration change

[Ning Shi]

In the job I'm implementing, there are a couple of configuration
variables that I wnat to change at runtime, such as rate limit at the
Kafka source. I know it's possible to use a control stream and join it
with the normal stream to configure things in certain operators, but
this doesn't work for the source. Is there any other way to configure
settings at runtime?

Thanks,

--
Ning

[Piotr Nowojski]

Hi,

Sorry but none that I’m aware of. As far as I know, the only way to dynamically configure Kafka source would be for you to copy and modify it’s code.

Piotrek

> On 6 Nov 2018, at 15:19, Ning Shi <[hidden email]> wrote:
>
> In the job I'm implementing, there are a couple of configuration
> variables that I wnat to change at runtime, such as rate limit at the
> Kafka source. I know it's possible to use a control stream and join it
> with the normal stream to configure things in certain operators, but
> this doesn't work for the source. Is there any other way to configure
> settings at runtime?
>
> Thanks,
>
> --
> Ning

[Nicos Maris]

Ning can you provide another example except for rate limiting?

On Tue, Nov 6, 2018, 6:20 PM Piotr Nowojski <[hidden email] wrote:

Hi,

Sorry but none that I’m aware of. As far as I know, the only way to dynamically configure Kafka source would be for you to copy and modify it’s code.

Piotrek

> On 6 Nov 2018, at 15:19, Ning Shi <[hidden email]> wrote:
>
> In the job I'm implementing, there are a couple of configuration
> variables that I wnat to change at runtime, such as rate limit at the
> Kafka source. I know it's possible to use a control stream and join it
> with the normal stream to configure things in certain operators, but
> this doesn't work for the source. Is there any other way to configure
> settings at runtime?
>
> Thanks,
>
> --
> Ning

[Ning Shi]

On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote:
> Ning can you provide another example except for rate limiting?

Our main use case and concern is rate limiting because without it we
could potentially overwhelm downstream systems (Cassandra) when the job
plays catch up or replay events from Kafka. The consequence of that is
that exceptions will be thrown in the Cassandra sink causing the whole
job to restart and end up in the same situation over and over.

With that said, we do have other use cases such as doing canary
deployment. We'd like to start processing events for a subset of users,
then expand it to all if things look good. Without live configuration,
we have to take a savepoint and restart the job, which will cause the
job to play catch up at the beginning, potentially overwhelming
downstream system without rate limiting.

Hope the use cases described above clarifies.

Thanks,

--
Ning

[Steven Wu]

for rate limiting, would quota at Kafka brokers help?

On Tue, Nov 6, 2018 at 10:29 AM Ning Shi <[hidden email]> wrote:

On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote:
> Ning can you provide another example except for rate limiting?

Our main use case and concern is rate limiting because without it we
could potentially overwhelm downstream systems (Cassandra) when the job
plays catch up or replay events from Kafka. The consequence of that is
that exceptions will be thrown in the Cassandra sink causing the whole
job to restart and end up in the same situation over and over.

With that said, we do have other use cases such as doing canary
deployment. We'd like to start processing events for a subset of users,
then expand it to all if things look good. Without live configuration,
we have to take a savepoint and restart the job, which will cause the
job to play catch up at the beginning, potentially overwhelming
downstream system without rate limiting.

Hope the use cases described above clarifies.

Thanks,

--
Ning

[Ning Shi]

> for rate limiting, would quota at Kafka brokers help?

Thanks, Steven. This looks very promising. I'll try it out.

--
Ning

[Elias Levy]

Also note that there is a pending PR to allow the Cassandra sink to back pressure, so that the cluster does not get overwhelmed. 

On Tue, Nov 6, 2018 at 12:46 PM Ning Shi <[hidden email]> wrote:

> for rate limiting, would quota at Kafka brokers help?

Thanks, Steven. This looks very promising. I'll try it out.

--
Ning

[Ning Shi]

> On Nov 6, 2018, at 4:22 PM, Elias Levy <[hidden email]> wrote:
>
> Also note that there is a pending PR to allow the Cassandra sink to back pressure, so that the cluster does not get overwhelmed.

Yes, I’ve been following the development on that pull request. Unfortunately, we have to go live very soon so we can’t wait to leverage that, but it’s definitely a very nice feature to have.

Thanks,

—
Ning