Hi,
In our application, we receive Orders and Prices via two KafkaSources. What
I want to do is to perform calculations for a given Order against a stream
of Prices for the same securityId, i.e. same identifier between the Order
and stream of Prices. Naturally this is a perfect fit for a KeyedStream
against the securityId for both Orders and Prices.
I have currently connected these two streams together and then processing by
ordersKeyStream.connect(pricesKeyStream).process(MyCoProcessFunction) and
this fits for 95% of the time. However part of my requirement is for
certain Orders, I need to be able to connect prices from a different
securityId (aka different key) to perform more calculations. From what I
can see, by the time I get to my CoProcessFunction, I am only able to see
the Orders and Prices for a single securityId, I won't be able to cross over
to another KeyedStream of Prices for me to perform this extra calculation.
In terms of this extra calculation, it is not a hard requirement to be able
to cross over to another KeyedStream of Prices, this is more ideal.
Things that I have thought about to get around this as it would be
acceptable to have a slightly older price for the securityId I require so:
1) I could connect to an external source of information to get this Price,
or
2) Periodically broadcast out a price that the ProcessFunction could consume
to perform this extra calculation.
This seems like something Flink should be easily able to handle, I just feel
as though I'm missing something here to allow this.
Just as something as a more non functional requirement. The number of
prices I receive per second can reach 10's of 000's per second, so that is
also something that I am very wary of as well
Is there anything that could be suggested to help me out on this?
Thanks in advance!
Scott
--
Sent from:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Locked
