Create a cluster inside Flink

[Simone Robutti]

Hello everyone,

I'm approaching a rather big and complex integration with an existing software and I would like to hear the opinion of more experienced users on how to tackle a few issues.

This software builds a cloud with its own logic. What I need is to keep these nodes as instances inside the TaskManagers and use these instances to perform operation with dedicated operators. I need to move tabular data back and forth from and to Flink's Datasets and be able to call methods on these instances.  

I would like to receive suggestions on how to implement this behaviour. First I thought about using Flink's actor system but I discovered it is not accessible. So I would like to understand how to properly create these instances (new thread inside a mapPartition?), how to call methods on them (create a custom context?) and convert data from a Dataset or a Table to the custom format of this software (this probably won't be much of a problem, I will write wrappers or at worst replicate the data).

Any suggestion is welcome.

Thanks,

Simone