Getting Window information from coGroup functin

[Sudan S]

Hi,

I have a usecase where i want to join two streams. I am using coGroup for this

KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey());
KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey());
leftSource.coGroup(rightSource).where(leftKey).equalTo(rightKey)
          .window(...)
          .apply()
          .addSink(...);

For apply method i'm using RichCoGroupFunction. I am not able to find access to Window object similar to

ProcessWindowFunction. I would be interested in extracting start time, end time and key of the window

Plz suggest if there are any alternatives

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

[Dawid Wysakowicz-2]

I am afraid there is no way to do that. At least I could not think of a way to do it.

Maybe @aljoscha cc'ed could help here.

On 29/05/2020 13:25, Sudan S wrote:

Hi,

I have a usecase where i want to join two streams. I am using coGroup for this

KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey());
KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey());
leftSource.coGroup(rightSource).where(leftKey).equalTo(rightKey)
          .window(...)
          .apply()
          .addSink(...);

For apply method i'm using RichCoGroupFunction. I am not able to find access to Window object similar to

ProcessWindowFunction. I would be interested in extracting start time, end time and key of the window

Plz suggest if there are any alternatives

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

I think here apply function would receive only the events but not necessarily a complete window at same time.

---

From: Dawid Wysakowicz
Sent: Thursday, June 04, 2020 13:39
To: Sudan S; [hidden email]
Cc: Aljoscha Krettek
Subject: Re: Getting Window information from coGroup functin

I am afraid there is no way to do that. At least I could not think of a way to do it.

Maybe @aljoscha cc'ed could help here.

On 29/05/2020 13:25, Sudan S wrote:

Hi,

I have a usecase where i want to join two streams. I am using coGroup for this

KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey());
KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey());
leftSource.coGroup(rightSource).where(leftKey).equalTo(rightKey)
          .window(...)
          .apply()
          .addSink(...);

For apply method i'm using RichCoGroupFunction. I am not able to find access to Window object similar to

ProcessWindowFunction. I would be interested in extracting start time, end time and key of the window

Plz suggest if there are any alternatives

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

However, You can evaluate your time window, get the information of window of which an event is a part from the context of processFunction or any RichFunction you are passing the events to. So, on each event arrival you will be able to check which window the element is part of and from Window object you can fetch the information of window, start time, end time etc.

---

From: Jaswin Shah <[hidden email]>
Sent: 04 June 2020 13:45
To: Dawid Wysakowicz <[hidden email]>; [hidden email] <[hidden email]>
Cc: Aljoscha Krettek <[hidden email]>
Subject: Re: Getting Window information from coGroup functin

 

I think here apply function would receive only the events but not necessarily a complete window at same time.

---

From: Dawid Wysakowicz
Sent: Thursday, June 04, 2020 13:39
To: Sudan S; [hidden email]
Cc: Aljoscha Krettek
Subject: Re: Getting Window information from coGroup functin

I am afraid there is no way to do that. At least I could not think of a way to do it.

Maybe @aljoscha cc'ed could help here.

On 29/05/2020 13:25, Sudan S wrote:

Hi,

I have a usecase where i want to join two streams. I am using coGroup for this

KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey());
KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey());
leftSource.coGroup(rightSource).where(leftKey).equalTo(rightKey)
          .window(...)
          .apply()
          .addSink(...);

For apply method i'm using RichCoGroupFunction. I am not able to find access to Window object similar to

ProcessWindowFunction. I would be interested in extracting start time, end time and key of the window

Plz suggest if there are any alternatives

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

[Arvid Heise-3]

Hi Sudan,

it seems to be unsupported directly.

You can have a hacky workaround by replicating apply[1] in your code and adjust the last line to call your CoGroupWindowFunction.

[1] https://github.com/apache/flink/blob/aedb4068408cfcad6f258526b00fcbff7f40fb82/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/CoGroupedStreams.java#L325-L359

On Thu, Jun 4, 2020 at 10:09 AM Dawid Wysakowicz <[hidden email]> wrote:

I am afraid there is no way to do that. At least I could not think of a way to do it.

Maybe @aljoscha cc'ed could help here.

On 29/05/2020 13:25, Sudan S wrote:

Hi,

I have a usecase where i want to join two streams. I am using coGroup for this

KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey());
KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey());
leftSource.coGroup(rightSource).where(leftKey).equalTo(rightKey)
          .window(...)
          .apply()
          .addSink(...);

For apply method i'm using RichCoGroupFunction. I am not able to find access to Window object similar to

ProcessWindowFunction. I would be interested in extracting start time, end time and key of the window

Plz suggest if there are any alternatives

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

--