Way to hold execution of one of the map operator in Co-FlatMaps

[Biplob Biswas]

Hi,

I was wondering whether it is possible to stop streaming data in from one of the map operators until some data arrives in the second map operator.

For ex,

if i have ds1.connect(ds2).map(new coflatmapper())

then, i need data to stop flowing from ds1 until some data arrives in ds2.

Is that possible? I know that's a problematic situation as ds1 may get overwhelmed with data if not processed quickly, but any work around would really helpful.


I was thinking of busy waiting, but I doubt that, the 2 maps work in different threads.

Regards
Biplob

[Aljoscha Krettek]

Hi,

the two map functions are called by the same thread, so waiting in one function would block all processing. What you could do is buffer elements from one input and only process them when an element arrives on the other input.

Cheers,

Aljoscha

On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <[hidden email]> wrote:

Hi,

I was wondering whether it is possible to stop streaming data in from one of
the map operators until some data arrives in the second map operator.

For ex,

if i have ds1.connect(ds2).map(new coflatmapper())

then, i need data to stop flowing from ds1 until some data arrives in ds2.

Is that possible? I know that's a problematic situation as ds1 may get
overwhelmed with data if not processed quickly, but any work around would
really helpful.


I was thinking of busy waiting, but I doubt that, the 2 maps work in
different threads.

Regards
Biplob



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Janardhan Reddy]

Hi,

Instead of buffering can we use event creation time and watermarks ?

On Mon, Jun 27, 2016 at 2:32 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

the two map functions are called by the same thread, so waiting in one function would block all processing. What you could do is buffer elements from one input and only process them when an element arrives on the other input.

Cheers,

Aljoscha

On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <[hidden email]> wrote:

Hi,

I was wondering whether it is possible to stop streaming data in from one of
the map operators until some data arrives in the second map operator.

For ex,

if i have ds1.connect(ds2).map(new coflatmapper())

then, i need data to stop flowing from ds1 until some data arrives in ds2.

Is that possible? I know that's a problematic situation as ds1 may get
overwhelmed with data if not processed quickly, but any work around would
really helpful.


I was thinking of busy waiting, but I doubt that, the 2 maps work in
different threads.

Regards
Biplob



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Aljoscha Krettek]

Maybe. But how do you mean, exactly?

On Mon, 27 Jun 2016 at 11:14 Janardhan Reddy <[hidden email]> wrote:

Hi,

Instead of buffering can we use event creation time and watermarks ?

On Mon, Jun 27, 2016 at 2:32 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

the two map functions are called by the same thread, so waiting in one function would block all processing. What you could do is buffer elements from one input and only process them when an element arrives on the other input.

Cheers,

Aljoscha

On Sun, 26 Jun 2016 at 13:36 Biplob Biswas <[hidden email]> wrote:

Hi,

I was wondering whether it is possible to stop streaming data in from one of
the map operators until some data arrives in the second map operator.

For ex,

if i have ds1.connect(ds2).map(new coflatmapper())

then, i need data to stop flowing from ds1 until some data arrives in ds2.

Is that possible? I know that's a problematic situation as ds1 may get
overwhelmed with data if not processed quickly, but any work around would
really helpful.


I was thinking of busy waiting, but I doubt that, the 2 maps work in
different threads.

Regards
Biplob



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Biplob Biswas]

Hi,

I was afraid of buffering because I am not sure when the second map function would get data, wouldn't the first map be flooded with data while the second map gets data?

Is it viable because the first stream is a stream of continuous data and the second stream gets data after some time and not as fast as the first one.

Thanks for replying though, will try that out.

Regards
Biplob Biswas

[Aljoscha Krettek]

Hi,

I might lead to flooding, yes. But I'm afraid it's the only way to go right now.

Cheers,

Aljoscha

On Mon, 27 Jun 2016 at 17:57 Biplob Biswas <[hidden email]> wrote:

Hi,

I was afraid of buffering because I am not sure when the second map function
would get data, wouldn't the first map be flooded with data while the second
map gets data?

Is it viable because the first stream is a stream of continuous data and the
second stream gets data after some time and not as fast as the first one.

Thanks for replying though, will try that out.

Regards
Biplob Biswas



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Way-to-hold-execution-of-one-of-the-map-operator-in-Co-FlatMaps-tp7689p7699.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.