Re: Understading Flink statefun deployment
Posted by
Igal Shilman on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Understading-Flink-statefun-deployment-tp35805p35828.html
Hi Francesco,
It is absolutely possible to deploy some functions as embedded and some as remote, and scale them independently, while technically being part of the same
stateful function application instance (I think that what you meant by "sharing the same master").
One possible way to do it in k8s, is to have separate deployments:
1) Embedded functions would be bundled with the Docker image that starts the Flink cluster. (flink-statefun docker image)
2) Remote functions would be packaged in a separate image deployed as a separate kubernetes deployment, and reachable via a k8s service.
For the second part checkout out the demo at the end of the keynote [1] and also [2][3][4]
Good luck,
Igal.
On Tue, Jun 9, 2020 at 10:50 AM Francesco Guardiani <
[hidden email]> wrote:
Hi everybody,
I'm quite new to Flink and Flink Statefun and I'm trying to understand the deployment techniques on k8s.
I wish to understand if it's feasible to deploy a statefun project separating the different functions on separate deployments (in order to have some functions as remote and some as embedded) all connected to the same master. The idea is that I can scale the deployments independently using the Kubernetes HPA and these instances cooperate automatically using the same master. For example, given a flow like kafka -> fn a -> fn b -> kafka:
* Remote function A (plus ingress) in deployment fn-a, where the function process is deployed as another container in the same pod
* embedded function B (plus egress) in deployment fn-b
* master deployment in flink-master
Does that make sense at all in Flink architecture? If it's feasible, do you have any example?
FG
--