About Memory Spilling to Disk in Flink

[Roc Marshal]

Hi, 

    Can someone tell me where flink uses memory spilling to write to disk? 

    Thank you.

Best, Roc.

 

[Guowei Ma]

Hi, Roc

Could you explain more about your question? 

On Wed, Mar 24, 2021 at 2:47 PM Roc Marshal <[hidden email]> wrote:

Hi, 

    Can someone tell me where flink uses memory spilling to write to disk? 

    Thank you.

Best, Roc.

 

[Roc Marshal]

Hi, Guowei Ma.

    As far as I know, flink writes some in-memory data to disk when memory is running low. I noticed that flink uses ExternalSorterBuilder for batch operations in the org.apache.flink.runtime.operator.sort package, but I'm curious to confirm if this technique is also used in stream mode.

    Thank you.

Best, 

Roc

At 2021-03-24 15:01:48, "Guowei Ma" <[hidden email]> wrote:

Hi, Roc

Could you explain more about your question? 

On Wed, Mar 24, 2021 at 2:47 PM Roc Marshal <[hidden email]> wrote:

Hi, 

    Can someone tell me where flink uses memory spilling to write to disk? 

    Thank you.

Best, Roc.

 

 

[Guowei Ma]

Hi, Roc

Thanks for your detailed explanation.

I could not find any "stream" operator that uses `ExternalSorterBuilder` by "find usage" of the IDEA. 

Best,

On Wed, Mar 24, 2021 at 3:27 PM Roc Marshal <[hidden email]> wrote:

Hi, Guowei Ma.

    As far as I know, flink writes some in-memory data to disk when memory is running low. I noticed that flink uses ExternalSorterBuilder for batch operations in the org.apache.flink.runtime.operator.sort package, but I'm curious to confirm if this technique is also used in stream mode.

    Thank you.

Best, 

Roc

At 2021-03-24 15:01:48, "Guowei Ma" <[hidden email]> wrote:

Hi, Roc

Could you explain more about your question? 

Best,

Guowei

On Wed, Mar 24, 2021 at 2:47 PM Roc Marshal <[hidden email]> wrote:

Hi, 

    Can someone tell me where flink uses memory spilling to write to disk? 

    Thank you.

Best, Roc.