using value that are not passed as parameter

[Michele Bertoni]

Hi everybody,
this question may sounds stupid, but i would like to have it clear

what happens if inside a dataset transformation (e.g. a map) I use something that is declared somewhere else, like a variable or a dataset, and not passed as broadcast dataset nor parameter in the constructor of a rich function?

for example

val a = 2
dataset.filter((v) => v._1 == a)

or

dataset1.flatmap((v) => /* something that uses dataset2 */)


thanks

[Stephan Ewen]

Hi Michele!

This may help explain things: https://cwiki.apache.org/confluence/display/FLINK/Variables+Closures+vs.+Broadcast+Variables

Greetings,

Stephan

On Wed, Jun 24, 2015 at 6:23 PM, Michele Bertoni <[hidden email]> wrote:

Hi everybody,
this question may sounds stupid, but i would like to have it clear

what happens if inside a dataset transformation (e.g. a map) I use something that is declared somewhere else, like a variable or a dataset, and not passed as broadcast dataset nor parameter in the constructor of a rich function?

for example

val a = 2
dataset.filter((v) => v._1 == a)

or

dataset1.flatmap((v) => /* something that uses dataset2 */)


thanks

[Michele Bertoni]

Yes it does!

Every time I find this kind of pages that explain internals they come out to be really useful

Il giorno 24/giu/2015, alle ore 18:35, Stephan Ewen <[hidden email]> ha scritto:

Hi Michele!

This may help explain things: <a href="https://cwiki.apache.org/confluence/display/FLINK/Variables&#43;Closures&#43;vs.&#43;Broadcast&#43;Variables" class="">https://cwiki.apache.org/confluence/display/FLINK/Variables+Closures+vs.+Broadcast+Variables

Greetings,

Stephan

On Wed, Jun 24, 2015 at 6:23 PM, Michele Bertoni <[hidden email]> wrote:

Hi everybody,
this question may sounds stupid, but i would like to have it clear

what happens if inside a dataset transformation (e.g. a map) I use something that is declared somewhere else, like a variable or a dataset, and not passed as broadcast dataset nor parameter in the constructor of a rich function?

for example

val a = 2
dataset.filter((v) => v._1 == a)

or

dataset1.flatmap((v) => /* something that uses dataset2 */)


thanks