(DEPRECATED) Apache Flink User Mailing List archive.

Concise example of how to deploy flink on Kubernetes

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
George Costea
Reply | Threaded
Open this post in threaded view
♦
♦
|

Concise example of how to deploy flink on Kubernetes

George Costea
Hi there,

Is there an example of how to deploy a flink cluster on Kubernetes?
I'd like to deploy the flink cluster, a kafka-broker, and then the
greeter example to give it a try.

Thanks,
George
Xingbo Huang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Concise example of how to deploy flink on Kubernetes

Xingbo Huang
Hi George,
Have you referred to the official document[1]?

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html

Best,
Xingbo

在 2020年11月21日星期六，George Costea <[hidden email]> 写道：
> Hi there,
>
> Is there an example of how to deploy a flink cluster on Kubernetes?
> I'd like to deploy the flink cluster, a kafka-broker, and then the
> greeter example to give it a try.
>
> Thanks,
> George
>
George Costea
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Concise example of how to deploy flink on Kubernetes

George Costea
Sorry.  Forgot to reply to all.

On Sun, Nov 22, 2020 at 9:24 PM George Costea <[hidden email]> wrote:

>
> Hi Xingbo,
>
> I’m interested in using stateful functions to build an application on Kubernetes. Don’t I need to deploy the flink cluster on Kubernetes first before deploying my stateful functions?
>
> Thanks,
> George
>
> On Sun, Nov 22, 2020 at 9:01 PM Xingbo Huang <[hidden email]> wrote:
>>
>> Hi, George
>>
>> >>>  Is zookeeper needed to run the flink cluster?
>> No. zookeeper is needed by the kafka.
>>
>> >>>  Or is it just the master and worker deployments?
>> This example uses the kafka connector, so some kafka and zookeeper configurations are added.
>>
>> >>>  Is the fink cluster included in the master and worker stateful functions?
>> In fact, I don’t really understand how this problem is related to stateful functions.
>>
>> Best,
>> Xingbo
>>
>> George Costea <[hidden email]> 于2020年11月21日周六 上午11:09写道：
>>>
>>> I looked at that.
>>>
>>> The docker-compose file for the greeter example deploys a worker, a
>>> master, zookeeper, and kafka.  Is zookeeper needed to run the flink
>>> cluster?  Or is it just the master and worker deployments?  Is the
>>> fink cluster included in the master and worker stateful functions?
>>>
>>> Thanks in advance.
>>>
>>> On Fri, Nov 20, 2020 at 9:32 PM Xingbo Huang <[hidden email]> wrote:
>>> >
>>> > Hi George,
>>> > Have you referred to the official document[1]?
>>> >
>>> > [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html
>>> >
>>> > Best,
>>> > Xingbo
>>> >
>>> > 在 2020年11月21日星期六，George Costea <[hidden email]> 写道：
>>> > > Hi there,
>>> > >
>>> > > Is there an example of how to deploy a flink cluster on Kubernetes?
>>> > > I'd like to deploy the flink cluster, a kafka-broker, and then the
>>> > > greeter example to give it a try.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Concise example of how to deploy flink on Kubernetes

Till Rohrmann
Hi George,

Here is some documentation about how to deploy a stateful function job [1]. In a nutshell, you need to deploy a Flink cluster on which you can run the stateful function job. This can either happen before (e.g. by spawning a session cluster on K8s [2]) or you can combine your job into a Docker image [3] which you can use to spin up a cluster.


Cheers,
Till

On Mon, Nov 23, 2020 at 3:39 PM George Costea <[hidden email]> wrote:
Sorry.  Forgot to reply to all.

On Sun, Nov 22, 2020 at 9:24 PM George Costea <[hidden email]> wrote:
>
> Hi Xingbo,
>
> I’m interested in using stateful functions to build an application on Kubernetes. Don’t I need to deploy the flink cluster on Kubernetes first before deploying my stateful functions?
>
> Thanks,
> George
>
> On Sun, Nov 22, 2020 at 9:01 PM Xingbo Huang <[hidden email]> wrote:
>>
>> Hi, George
>>
>> >>>  Is zookeeper needed to run the flink cluster?
>> No. zookeeper is needed by the kafka.
>>
>> >>>  Or is it just the master and worker deployments?
>> This example uses the kafka connector, so some kafka and zookeeper configurations are added.
>>
>> >>>  Is the fink cluster included in the master and worker stateful functions?
>> In fact, I don’t really understand how this problem is related to stateful functions.
>>
>> Best,
>> Xingbo
>>
>> George Costea <[hidden email]> 于2020年11月21日周六 上午11:09写道：
>>>
>>> I looked at that.
>>>
>>> The docker-compose file for the greeter example deploys a worker, a
>>> master, zookeeper, and kafka.  Is zookeeper needed to run the flink
>>> cluster?  Or is it just the master and worker deployments?  Is the
>>> fink cluster included in the master and worker stateful functions?
>>>
>>> Thanks in advance.
>>>
>>> On Fri, Nov 20, 2020 at 9:32 PM Xingbo Huang <[hidden email]> wrote:
>>> >
>>> > Hi George,
>>> > Have you referred to the official document[1]?
>>> >
>>> > [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html
>>> >
>>> > Best,
>>> > Xingbo
>>> >
>>> > 在 2020年11月21日星期六，George Costea <[hidden email]> 写道：
>>> > > Hi there,
>>> > >
>>> > > Is there an example of how to deploy a flink cluster on Kubernetes?
>>> > > I'd like to deploy the flink cluster, a kafka-broker, and then the
>>> > > greeter example to give it a try.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
George Costea
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Concise example of how to deploy flink on Kubernetes

George Costea
Thank you. This is very helpful. 

On Mon, Nov 23, 2020 at 9:46 AM Till Rohrmann <[hidden email]> wrote:
Hi George,

Here is some documentation about how to deploy a stateful function job [1]. In a nutshell, you need to deploy a Flink cluster on which you can run the stateful function job. This can either happen before (e.g. by spawning a session cluster on K8s [2]) or you can combine your job into a Docker image [3] which you can use to spin up a cluster.


Cheers,
Till

On Mon, Nov 23, 2020 at 3:39 PM George Costea <[hidden email]> wrote:
Sorry.  Forgot to reply to all.

On Sun, Nov 22, 2020 at 9:24 PM George Costea <[hidden email]> wrote:
>
> Hi Xingbo,
>
> I’m interested in using stateful functions to build an application on Kubernetes. Don’t I need to deploy the flink cluster on Kubernetes first before deploying my stateful functions?
>
> Thanks,
> George
>
> On Sun, Nov 22, 2020 at 9:01 PM Xingbo Huang <[hidden email]> wrote:
>>
>> Hi, George
>>
>> >>>  Is zookeeper needed to run the flink cluster?
>> No. zookeeper is needed by the kafka.
>>
>> >>>  Or is it just the master and worker deployments?
>> This example uses the kafka connector, so some kafka and zookeeper configurations are added.
>>
>> >>>  Is the fink cluster included in the master and worker stateful functions?
>> In fact, I don’t really understand how this problem is related to stateful functions.
>>
>> Best,
>> Xingbo
>>
>> George Costea <[hidden email]> 于2020年11月21日周六 上午11:09写道：
>>>
>>> I looked at that.
>>>
>>> The docker-compose file for the greeter example deploys a worker, a
>>> master, zookeeper, and kafka.  Is zookeeper needed to run the flink
>>> cluster?  Or is it just the master and worker deployments?  Is the
>>> fink cluster included in the master and worker stateful functions?
>>>
>>> Thanks in advance.
>>>
>>> On Fri, Nov 20, 2020 at 9:32 PM Xingbo Huang <[hidden email]> wrote:
>>> >
>>> > Hi George,
>>> > Have you referred to the official document[1]?
>>> >
>>> > [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html
>>> >
>>> > Best,
>>> > Xingbo
>>> >
>>> > 在 2020年11月21日星期六，George Costea <[hidden email]> 写道：
>>> > > Hi there,
>>> > >
>>> > > Is there an example of how to deploy a flink cluster on Kubernetes?
>>> > > I'd like to deploy the flink cluster, a kafka-broker, and then the
>>> > > greeter example to give it a try.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
Igal Shilman
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Concise example of how to deploy flink on Kubernetes

Igal Shilman
Hi George,
Specifically for StateFun, we have the following Helm charts [1] to help you deploy Stateful Functions on k8s.

The greeter example's docker-compose file also includes Kafka (and hence Zookeeper).
Indeed the Flink cluster is "included" in the master/worker stateful functions docker image. 
Starting the image with the environment variable  ROLE=master, would start a JobManager, while starting it with ROLE=worker will start a TaskManager.


Good luck,
Igal.



On Tue, Nov 24, 2020 at 11:25 PM George Costea <[hidden email]> wrote:
Thank you. This is very helpful. 

On Mon, Nov 23, 2020 at 9:46 AM Till Rohrmann <[hidden email]> wrote:
Hi George,

Here is some documentation about how to deploy a stateful function job [1]. In a nutshell, you need to deploy a Flink cluster on which you can run the stateful function job. This can either happen before (e.g. by spawning a session cluster on K8s [2]) or you can combine your job into a Docker image [3] which you can use to spin up a cluster.


Cheers,
Till

On Mon, Nov 23, 2020 at 3:39 PM George Costea <[hidden email]> wrote:
Sorry.  Forgot to reply to all.

On Sun, Nov 22, 2020 at 9:24 PM George Costea <[hidden email]> wrote:
>
> Hi Xingbo,
>
> I’m interested in using stateful functions to build an application on Kubernetes. Don’t I need to deploy the flink cluster on Kubernetes first before deploying my stateful functions?
>
> Thanks,
> George
>
> On Sun, Nov 22, 2020 at 9:01 PM Xingbo Huang <[hidden email]> wrote:
>>
>> Hi, George
>>
>> >>>  Is zookeeper needed to run the flink cluster?
>> No. zookeeper is needed by the kafka.
>>
>> >>>  Or is it just the master and worker deployments?
>> This example uses the kafka connector, so some kafka and zookeeper configurations are added.
>>
>> >>>  Is the fink cluster included in the master and worker stateful functions?
>> In fact, I don’t really understand how this problem is related to stateful functions.
>>
>> Best,
>> Xingbo
>>
>> George Costea <[hidden email]> 于2020年11月21日周六 上午11:09写道：
>>>
>>> I looked at that.
>>>
>>> The docker-compose file for the greeter example deploys a worker, a
>>> master, zookeeper, and kafka.  Is zookeeper needed to run the flink
>>> cluster?  Or is it just the master and worker deployments?  Is the
>>> fink cluster included in the master and worker stateful functions?
>>>
>>> Thanks in advance.
>>>
>>> On Fri, Nov 20, 2020 at 9:32 PM Xingbo Huang <[hidden email]> wrote:
>>> >
>>> > Hi George,
>>> > Have you referred to the official document[1]?
>>> >
>>> > [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html
>>> >
>>> > Best,
>>> > Xingbo
>>> >
>>> > 在 2020年11月21日星期六，George Costea <[hidden email]> 写道：
>>> > > Hi there,
>>> > >
>>> > > Is there an example of how to deploy a flink cluster on Kubernetes?
>>> > > I'd like to deploy the flink cluster, a kafka-broker, and then the
>>> > > greeter example to give it a try.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
Free forum by Nabble Edit this page