Multi threaded operators?

[Alexander Smirnov]

Hi,

I have a co-flatmap function which reads data from external DB on specific events.

The API for the DB layer is homegrown and it uses multiple threads to speed up reading. 

Can it cause any problems if I use the multithreading API in the flatmap1 function? Is it allowed in Flink?

Or, maybe I should employ better approaches for that. May be async I/O? 

Thank you,

Alex

[Jörn Franke]

I would disable it if possible and use the Flink  parallism. The threading might work but can create operational issues depending on how you configure your resource manager.

> On 23. Apr 2018, at 11:54, Alexander Smirnov <[hidden email]> wrote:
>
> Hi,
>
> I have a co-flatmap function which reads data from external DB on specific events.
> The API for the DB layer is homegrown and it uses multiple threads to speed up reading.
>
> Can it cause any problems if I use the multithreading API in the flatmap1 function? Is it allowed in Flink?
>
> Or, maybe I should employ better approaches for that. May be async I/O?
>
> Thank you,
> Alex

[Lasse Nedergaard]

Hi.
What kind of problems and what configuration should we be aware of.?

Med venlig hilsen / Best regards
Lasse Nedergaard

> Den 23. apr. 2018 kl. 13.44 skrev Jörn Franke <[hidden email]>:
>
> I would disable it if possible and use the Flink  parallism. The threading might work but can create operational issues depending on how you configure your resource manager.
>
>> On 23. Apr 2018, at 11:54, Alexander Smirnov <[hidden email]> wrote:
>>
>> Hi,
>>
>> I have a co-flatmap function which reads data from external DB on specific events.
>> The API for the DB layer is homegrown and it uses multiple threads to speed up reading.
>>
>> Can it cause any problems if I use the multithreading API in the flatmap1 function? Is it allowed in Flink?
>>
>> Or, maybe I should employ better approaches for that. May be async I/O?
>>
>> Thank you,
>> Alex

[Ken Krugler]

Hi Alex,

Given that you’re hitting a DB, the approach of using multi-threaded access from a CoFlatMapFunction or AsyncFunction makes sense - you don’t want to try to abuse Flink’s parallelism.

I’ve done it both ways, so either is an option.

If you use an AsyncFunction, you get the benefit of correct checkpointing for in-flight data, and it’s significantly easier to use.

With a CoFlatMapFunction, I wound up having to put the async results into a local queue, and then send tickler records to the other method that triggered pulling results out of this queue an calling the collector to generate output, as it’s not clear that you can safely hang onto a regular collector and call it asynchronously when threads complete.

— Ken

> On Apr 23, 2018, at 10:59 AM, Lasse Nedergaard <[hidden email]> wrote:
>
> Hi.
> What kind of problems and what configuration should we be aware of.?
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
>> Den 23. apr. 2018 kl. 13.44 skrev Jörn Franke <[hidden email]>:
>>
>> I would disable it if possible and use the Flink  parallism. The threading might work but can create operational issues depending on how you configure your resource manager.
>>
>>> On 23. Apr 2018, at 11:54, Alexander Smirnov <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I have a co-flatmap function which reads data from external DB on specific events.
>>> The API for the DB layer is homegrown and it uses multiple threads to speed up reading.
>>>
>>> Can it cause any problems if I use the multithreading API in the flatmap1 function? Is it allowed in Flink?
>>>
>>> Or, maybe I should employ better approaches for that. May be async I/O?
>>>
>>> Thank you,
>>> Alex

--------------------------------------------
http://about.me/kkrugler
[phone]