Reading static data

[Mohit Anchlia]

What is the best way to read a map of lookup data? This lookup data is like a small short lived data that is available in transformation to do things like filtering, additional augmentation of data etc.

[Timo Walther]

Hi Mohit,

do you plan to implement a batch or streaming job? If it is a streaming
job: You can use a connected stream (see [1], Slide 34). The static data
is one side of the stream that could be updated from time to time and
will always propagated (using a broadcast()) to all workers that do
filtering, augmentation etc.

[1] http://training.data-artisans.com/dataStream/1-intro.html

I hope this helps.

Timo


Am 13.07.17 um 02:16 schrieb Mohit Anchlia:
> What is the best way to read a map of lookup data? This lookup data is
> like a small short lived data that is available in transformation to
> do things like filtering, additional augmentation of data etc.

[Mohit Anchlia]

Is there a way to accomplish this for the batch operations?

On Thu, Jul 13, 2017 at 4:59 AM, Timo Walther <[hidden email]> wrote:

Hi Mohit,

do you plan to implement a batch or streaming job? If it is a streaming job: You can use a connected stream (see [1], Slide 34). The static data is one side of the stream that could be updated from time to time and will always propagated (using a broadcast()) to all workers that do filtering, augmentation etc.

[1] http://training.data-artisans.com/dataStream/1-intro.html

I hope this helps.

Timo


Am 13.07.17 um 02:16 schrieb Mohit Anchlia:

What is the best way to read a map of lookup data? This lookup data is like a small short lived data that is available in transformation to do things like filtering, additional augmentation of data etc.

[Fabian Hueske-2]

You could either use a broadcast variable [1] or the distributed cache [2].

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html#broadcast-variables
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html#distributed-cache

2017-07-14 20:18 GMT+02:00 Mohit Anchlia <[hidden email]>:

Is there a way to accomplish this for the batch operations?

On Thu, Jul 13, 2017 at 4:59 AM, Timo Walther <[hidden email]> wrote:

Hi Mohit,

do you plan to implement a batch or streaming job? If it is a streaming job: You can use a connected stream (see [1], Slide 34). The static data is one side of the stream that could be updated from time to time and will always propagated (using a broadcast()) to all workers that do filtering, augmentation etc.

[1] http://training.data-artisans.com/dataStream/1-intro.html

I hope this helps.

Timo


Am 13.07.17 um 02:16 schrieb Mohit Anchlia:

What is the best way to read a map of lookup data? This lookup data is like a small short lived data that is available in transformation to do things like filtering, additional augmentation of data etc.