How to trigger a Global Window with a different Message from the window message

[Oliver Buckley-Salmon]

Hi,

 

I have a Flink job where I receive a stream of AggregationKeys, stored in BroadcastState which I join in a Tuple2 with a stream of RiskMeasureMessages, which I then wish to aggregate in a Window.

The RiskMeasureMessages are bounded by CalcStart and CalcEnd messages which come on separate Kafka topics, we can ignore CalcStart but need to emit the aggregated results after receiving CalcEnd. The CalcEnd messages are unkeyed.

My issue is how to get the CalcEnd message to the Trigger to use in the onElement() method, the only way I can see to do it is to create a co-stream but I don’t see how to Window and Trigger that, or merge the co-stream into Tuple3 wich would have Null for the 3^rd element all the time except when EndCalc is received.

Are there better ways for doing this?

I’m running Flink 1.7.1 using the Java API.

Thanks in advance for your help.

 

Kind regards,
Oliver Buckley-Salmon

____________________________________________________

 

 

---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.

[Kien Truong]

Hi Oliver,

Try replacing Global Window with a KeyedProcessFunction.

Store all the item received between CalcStart and CalcEnd inside a ListState the process them when CalcEnd is received.

Regards,

Kien

On 1/17/2019 1:06 AM, Oliver Buckley-Salmon wrote:

Hi,

 

I have a Flink job where I receive a stream of AggregationKeys, stored in BroadcastState which I join in a Tuple2 with a stream of RiskMeasureMessages, which I then wish to aggregate in a Window.

The RiskMeasureMessages are bounded by CalcStart and CalcEnd messages which come on separate Kafka topics, we can ignore CalcStart but need to emit the aggregated results after receiving CalcEnd. The CalcEnd messages are unkeyed.

My issue is how to get the CalcEnd message to the Trigger to use in the onElement() method, the only way I can see to do it is to create a co-stream but I don’t see how to Window and Trigger that, or merge the co-stream into Tuple3 wich would have Null for the 3^rd element all the time except when EndCalc is received.

Are there better ways for doing this?

I’m running Flink 1.7.1 using the Java API.

Thanks in advance for your help.

 

Kind regards,
Oliver Buckley-Salmon

____________________________________________________

 

 

---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to https://www.db.com/disclosures for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy.