Windows on SinkFunctions

[Sidney Feiner]

Hey,

I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.

For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.

Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline? 

Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

[tison]

Hi Sidney,

For the case, you can exactly write

stream.

  ...

  .window()

  .apply()

  .addSink()

Operator chain will chain these operators into one so that you don't have to worry about the efficiency.

Sidney Feiner <[hidden email]> 于2020年3月22日周日 下午10:03写道：

Hey,

I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.

For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.

Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline? 

Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

[Sidney Feiner]

Thanks!

What am I supposed to put in the apply/process function for the sink to be invoked on a List of items?

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

---

From: tison <[hidden email]>
Sent: Sunday, March 22, 2020 4:19 PM
To: Sidney Feiner <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: Windows on SinkFunctions

 

Hi Sidney,

For the case, you can exactly write

stream.

  ...

  .window()

  .apply()

  .addSink()

Operator chain will chain these operators into one so that you don't have to worry about the efficiency.

Sidney Feiner <[hidden email]> 于2020年3月22日周日 下午10:03写道：

Hey,

I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.

For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.

Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline? 

Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

[rmetzger0]

Hey,

In your original email, you wrote: 

 Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

You can also send the data of an operator to multiple sinks

Source --> MyComputationProcessFunction --> DataBatcher --> BatchedSink
                                         \
                                          \--> StreamingSink

(also here if it renders weirdly in email: https://gist.github.com/rmetzger/9ef311c2926a82fdba2b2b7af9ad65d6 )

In this example "MyComputationProcessFunction" is sending the data to two downstream operators: DataBatcher and StreamingSink.

For the "DataBatcher", you can also consider building something yourself with ProcessFunction. I would just collect the data in a List, and emit it once the list reached 500 elements, or a custom 5 minute trigger has triggered.

On Sun, Mar 29, 2020 at 12:07 PM Sidney Feiner <[hidden email]> wrote:

Thanks!

What am I supposed to put in the apply/process function for the sink to be invoked on a List of items?

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

---

From: tison <[hidden email]>
Sent: Sunday, March 22, 2020 4:19 PM
To: Sidney Feiner <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: Windows on SinkFunctions

 

Hi Sidney,

For the case, you can exactly write

stream.

  ...

  .window()

  .apply()

  .addSink()

Operator chain will chain these operators into one so that you don't have to worry about the efficiency.

Best,

tison.

Sidney Feiner <[hidden email]> 于2020年3月22日周日 下午10:03写道：

Hey,

I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.

For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.

Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline? 

Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)

Sidney Feiner / Data Platform Developer

M: +972.528197720 / Skype: sidney.feiner.startapp