fliter and flatMap operation VS only a flatMap operation

[Soheil Pourbafrani]

Hi,

In case we need to filter operation followed by a transformation, which one is more efficient in Flink, applying the filter operation first and then a flatMap operation separately OR using only a flatMap operation that internally includes the filter logic, too?

best

Soheil

[Tzu-Li (Gordon) Tai]

Hi,

If your filter and flatMap operators are chained, then the performance difference should not be noticeable.
If a shuffle (i.e. a keyBy operation) occurs after the filter and before the flatMap, then applying the filter first will be more efficient.

Cheers,
Gordon

On Thu, Jan 30, 2020 at 4:03 AM Soheil Pourbafrani <[hidden email]> wrote:

Hi,

In case we need to filter operation followed by a transformation, which one is more efficient in Flink, applying the filter operation first and then a flatMap operation separately OR using only a flatMap operation that internally includes the filter logic, too?

best

Soheil