Handling validations/errors in the flink job

[sagar]

Hi Team,

I am creating a flink job with DataStream API and batch mode.

It is having 5 different bounded sources and I need to perform some business operations on it like joining , aggregating etc.

 

I am using a CoGroup operator to join two streams as it serves as a left join. So when keys are present in both the stream, I am processing and moving ahead.

But when there is only one key present I need to send it as an error.

 

Some operators like Process have side output features, but CoGroup doesn't have that feature.

 

In order to report missing data to different stream, I am planning to create one common error handling stream and at each CoGroup operation I am planning to write it to error stream by using Split operator after CoGroup

 

Let me know if that is the correct way of handling the errors?

--

[sagar]

Hi Team, any answer for my below question?

On Wed, Jan 20, 2021 at 9:20 PM sagar <[hidden email]> wrote:

Hi Team,

I am creating a flink job with DataStream API and batch mode.

It is having 5 different bounded sources and I need to perform some business operations on it like joining , aggregating etc.

 

I am using a CoGroup operator to join two streams as it serves as a left join. So when keys are present in both the stream, I am processing and moving ahead.

But when there is only one key present I need to send it as an error.

 

Some operators like Process have side output features, but CoGroup doesn't have that feature.

 

In order to report missing data to different stream, I am planning to create one common error handling stream and at each CoGroup operation I am planning to write it to error stream by using Split operator after CoGroup

 

Let me know if that is the correct way of handling the errors?

--

---Regards---

  Sagar Bandal

This is confidential mail ,All Rights are Reserved.If you are not the intended recipient please ignore this email.

--

[Matthias]

Hi Sagar,

have you had a look at CoProcessFunction [1]? CoProcessFunction enables you to join two streams into one and also provide context to use SideOutput [2].

Best,
Matthias

[1] https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/CoProcessFunction.java

[2] https://ci.apache.org/projects/flink/flink-docs-master/learn-flink/event_driven.html#side-outputs

On Wed, Jan 20, 2021 at 4:51 PM sagar <[hidden email]> wrote:

Hi Team,

I am creating a flink job with DataStream API and batch mode.

It is having 5 different bounded sources and I need to perform some business operations on it like joining , aggregating etc.

 

I am using a CoGroup operator to join two streams as it serves as a left join. So when keys are present in both the stream, I am processing and moving ahead.

But when there is only one key present I need to send it as an error.

 

Some operators like Process have side output features, but CoGroup doesn't have that feature.

 

In order to report missing data to different stream, I am planning to create one common error handling stream and at each CoGroup operation I am planning to write it to error stream by using Split operator after CoGroup

 

Let me know if that is the correct way of handling the errors?

--

---Regards---

  Sagar Bandal

This is confidential mail ,All Rights are Reserved.If you are not the intended recipient please ignore this email.