Question about Memory Manage in the Streaming mode

[Tao Meng]

Hi all,

  I have some questions about memory management in the Streaming mode.

  Do the Streaming jobs use the memory management module ?

If they don't,  for what considerations do not ?  Because Data exchange is too frequent ?

Is there a plan to let streaming job use it ?

Thanks a lot.

[Timo Walther]

Hi Tao,

no, streaming jobs do not use managed memory yet. Managed memory is useful for sorting, joining and grouping bounded data. Unbounded stream do not need that.

It could be used in the future e.g. to store state or for new operators, but is this is not on the roadmap so far.

Regards,
Timo


Am 15/12/16 um 10:30 schrieb Tao Meng:

Hi all,

  I have some questions about memory management in the Streaming mode.

  Do the Streaming jobs use the memory management module ?

If they don't,  for what considerations do not ?  Because Data exchange is too frequent ?

Is there a plan to let streaming job use it ?

Thanks a lot.

[Tao Meng]

Thanks a lot.

On 12月 15 2016, at 5:39 下午, Timo Walther <[hidden email]> wrote:

Hi Tao,

no, streaming jobs do not use managed memory yet. Managed memory is useful for sorting, joining and grouping bounded data. Unbounded stream do not need that.

It could be used in the future e.g. to store state or for new operators, but is this is not on the roadmap so far.

Regards,
Timo


Am 15/12/16 um 10:30 schrieb Tao Meng:

Hi all,

  I have some questions about memory management in the Streaming mode.

  Do the Streaming jobs use the memory management module ?

If they don't,  for what considerations do not ?  Because Data exchange is too frequent ?

Is there a plan to let streaming job use it ?

Thanks a lot.

[Stephan Ewen]

Hi!

I do slightly disagree with Timo. Custom memory management is always useful, also in the Streaming API. It makes execution more robust.

If you use RocksDB as a state backend, you get memory management from RocksDB - effectively all your program key/value state is off-heap.

Flink's own state backends have not yet implemented custom memory management (it is quite a bit more complex in a true streaming environment than in batch), but it will come as a feature (though not officially tracked as a jira).

Stephan

On Thu, Dec 15, 2016 at 10:43 AM, Tao Meng <[hidden email]> wrote:

Thanks a lot.

On 12月 15 2016, at 5:39 下午, Timo Walther <[hidden email]> wrote:

Hi Tao,

no, streaming jobs do not use managed memory yet. Managed memory is useful for sorting, joining and grouping bounded data. Unbounded stream do not need that.

It could be used in the future e.g. to store state or for new operators, but is this is not on the roadmap so far.

Regards,
Timo


Am 15/12/16 um 10:30 schrieb Tao Meng:

Hi all,

  I have some questions about memory management in the Streaming mode.

  Do the Streaming jobs use the memory management module ?

If they don't,  for what considerations do not ?  Because Data exchange is too frequent ?

Is there a plan to let streaming job use it ?

Thanks a lot.