Best practices for assigning operator UIDs

[Jared Stehler]

I have some confusion around how best to assign UIDs to operators. The documentation states simply that they are important, but stops short of recommending what if any stateful information should go into the name. For example, if the same code is used to create two separate job graphs, should the operator UIDs contain information specific to each job instance?

Example: for a kafka source, should the UID contain the subscribed topic name / pattern?

Example: if I have a custom mapping function which reshapes a JSON field based on an expression, should that expression (or a hash thereof) be contained in that operator’s UID?

Basically, is the UID similar to overriding hashcode and equals for POJOs, or is it more like overriding serializableVersionUID?


--
Jared Stehler
Chief Architect - Intellify Learning
o: 617.701.6330 x703

[Gyula Fóra]

Hi Jared,

The only thing that matters is that UIDs are unique within one JobGraph. Its completely fine to use the same uids in two separate jobs.

Beyond this I would go with simple uids that dont contain parts of the logic, because maybe you want to change the logic (expression, or add new topics) and still keep the state.

Hope this helps :)
Gyula

On Sun, Jul 9, 2017, 18:36 Jared Stehler <[hidden email]> wrote:

I have some confusion around how best to assign UIDs to operators. The documentation states simply that they are important, but stops short of recommending what if any stateful information should go into the name. For example, if the same code is used to create two separate job graphs, should the operator UIDs contain information specific to each job instance?

Example: for a kafka source, should the UID contain the subscribed topic name / pattern?

Example: if I have a custom mapping function which reshapes a JSON field based on an expression, should that expression (or a hash thereof) be contained in that operator’s UID?

Basically, is the UID similar to overriding hashcode and equals for POJOs, or is it more like overriding serializableVersionUID?


--
Jared Stehler
Chief Architect - Intellify Learning
o: 617.701.6330 x703