Structuring the codebase for a complex Flink application

[Oytun Tez]

Hi there,

We are picking up more speed with Flink and already run several very simple operators in the production.

Our code base is getting more complicated and before things go out of hand, I'd like to understand how to think about the structure of this kind of a program.

I listed some topics, points, issues, things to think about here: https://atlas.mindmup.com/2019/04/ce30a3e0639211e9aadea960e2b9cac4/free_mind_map/index.html

Also an image version here:

How do you organize your codebase when you (are going to) have tens of operators, lots of business logic going on all over the place etc? 

In our case, we have multiple business cases (this is fraud, that person provides good quality work, this project is very important etc kind of conclusions), using multiple sources, some independent from each other (but can later be connected to another logic), some share multiple elements with each other (source, some output streams, sometimes side outputs, broadcasts etc)...

I would love to see some complexish codebases with hundreds of entities in different roles... Any guidance?

This article helped me a lot to bring together possible use cases of this system. Now I want to get done with my general confusion on how to represent the use cases and its composing elements together properly, with reusability and Flink-specific constraints in mind.

Thank you for the insights!

Oytun

[Oytun Tez]

Up 🙈🙊😇💅😰

On Sat, Apr 20, 2019 at 1:48 PM Oytun Tez <[hidden email]> wrote:

Hi there,

We are picking up more speed with Flink and already run several very simple operators in the production.

Our code base is getting more complicated and before things go out of hand, I'd like to understand how to think about the structure of this kind of a program.

I listed some topics, points, issues, things to think about here: https://atlas.mindmup.com/2019/04/ce30a3e0639211e9aadea960e2b9cac4/free_mind_map/index.html

Also an image version here:

How do you organize your codebase when you (are going to) have tens of operators, lots of business logic going on all over the place etc? 

In our case, we have multiple business cases (this is fraud, that person provides good quality work, this project is very important etc kind of conclusions), using multiple sources, some independent from each other (but can later be connected to another logic), some share multiple elements with each other (source, some output streams, sometimes side outputs, broadcasts etc)...

I would love to see some complexish codebases with hundreds of entities in different roles... Any guidance?

This article helped me a lot to bring together possible use cases of this system. Now I want to get done with my general confusion on how to represent the use cases and its composing elements together properly, with reusability and Flink-specific constraints in mind.

Thank you for the insights!

Oytun