Limit buffer size for a job

[Andrew Ge Wu]

Hi guys

We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?

Any suggestion? thanks!


Andrew
--
Confidentiality Notice: This e-mail transmission may contain confidential
or legally privileged information that is intended only for the individual
or entity named in the e-mail address. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution, or reliance upon the contents of this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete all copies
of this message.

[Ufuk Celebi]

Hey Andrew,

take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency

Does this help?

– Ufuk

On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <[hidden email]> wrote:

> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

[Stephan Ewen]

Hi!

Ufuk's suggestion explains how to buffer less between Flink operators.

Is that what you were looking for, or are you looking for a way to fetch more fine grained in the source from the message queue?

What type of source are you using?

Greetings,

Stephan

On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Andrew,

take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency

Does this help?

– Ufuk

On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <[hidden email]> wrote:
> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

[Andrew Ge Wu]

Thanks guys for the explanation, I will give it a try.

After all buffers are filled, the back pressure did it’s job, it works so far so good, but I will defiantly give a try to control the latency.

Thanks again!

Andrew

On 11 Apr 2016, at 18:19, Stephan Ewen <[hidden email]> wrote:

Hi!

Ufuk's suggestion explains how to buffer less between Flink operators.

Is that what you were looking for, or are you looking for a way to fetch more fine grained in the source from the message queue?

What type of source are you using?

Greetings,

Stephan

On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Andrew,

take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency

Does this help?

– Ufuk

On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <[hidden email]> wrote:
> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

Confidentiality Notice: This e-mail transmission may contain confidential or legally privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or reliance upon the contents of this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message.

[Stephan Ewen]

You can reduce Flink's internal network buffering by adjusting the total number of network buffers.

See https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-the-network-buffers

That should also make a difference.

On Wed, Apr 13, 2016 at 3:57 PM, Andrew Ge Wu <[hidden email]> wrote:

Thanks guys for the explanation, I will give it a try.

After all buffers are filled, the back pressure did it’s job, it works so far so good, but I will defiantly give a try to control the latency.

Thanks again!

Andrew

On 11 Apr 2016, at 18:19, Stephan Ewen <[hidden email]> wrote:

Hi!

Ufuk's suggestion explains how to buffer less between Flink operators.

Is that what you were looking for, or are you looking for a way to fetch more fine grained in the source from the message queue?

What type of source are you using?

Greetings,

Stephan

On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Andrew,

take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency

Does this help?

– Ufuk

On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <[hidden email]> wrote:
> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

Confidentiality Notice: This e-mail transmission may contain confidential or legally privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or reliance upon the contents of this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message.