connecting two streams flink

Team,

I have two kafka consumer for same topic and want to join second stream to first after couple of subtasks computation in the first stream then validate the record . KT - C1 ,C2

KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to DB
-C2 - Process --------------------------

if record is invalid then save it to Error topic .

How do I merge these two streams one(c2) is fast and other(c1) is little slow (two levels of computation) ?

Same record is flowing from C1-Flatmap-FlatMap and other consumer C2 . I have to validate that record based on the rules.
Please find the attached image herewith reference. 

--

[miki haiat]

If c1 and c2 are  listing   to the same topic they will  consume the same data .

so i cant understand this 

 these two streams one(c2) is fast and other(c1) 

 

On Tue, Jan 29, 2019 at 2:44 PM Selvaraj chennappan <[hidden email]> wrote:

Team,

I have two kafka consumer for same topic and want to join second stream to first after couple of subtasks computation in the first stream then validate the record . KT - C1 ,C2

KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to DB
-C2 - Process --------------------------

if record is invalid then save it to Error topic .

How do I merge these two streams one(c2) is fast and other(c1) is little slow (two levels of computation) ?

Same record is flowing from C1-Flatmap-FlatMap and other consumer C2 . I have to validate that record based on the rules.
Please find the attached image herewith reference. 

--

I have  pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB .

Before reaching to DB task lots of transformation is applied on the pipeline  Therefore want to validate the record with raw json message which is available in kafka  with the transformed record.

Hence I want to know How to do that in flink. 

On Tue, Jan 29, 2019 at 6:43 PM miki haiat <[hidden email]> wrote:

If c1 and c2 are  listing   to the same topic they will  consume the same data .

so i cant understand this 

 these two streams one(c2) is fast and other(c1) 

 

On Tue, Jan 29, 2019 at 2:44 PM Selvaraj chennappan <[hidden email]> wrote:

Team,

I have two kafka consumer for same topic and want to join second stream to first after couple of subtasks computation in the first stream then validate the record . KT - C1 ,C2

KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to DB
-C2 - Process --------------------------

if record is invalid then save it to Error topic .

How do I merge these two streams one(c2) is fast and other(c1) is little slow (two levels of computation) ?

Same record is flowing from C1-Flatmap-FlatMap and other consumer C2 . I have to validate that record based on the rules.
Please find the attached image herewith reference. 

--

--