FLIP-6 Docker / Kubernetes

[anaray]

Hi,

I was reading *Docker/Kubernetes* section of FLIP-6
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077).
I have question regarding point 1 ie "A job-specific image with the job
libraries, configured to launch the job (with a JobManager)"

How does job submission happens here ? Do we need write a script to submit
the job?





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Fabian Hueske-2]

Hi,

Most, but not all, of the FLIP-6 features will be released with Flink 1.5.0.

I'm not sure if this deployment mode will be fully supported in 1.5.0. Gary (in CC) might know details here.

Anyway, the deployment would work by starting the image using regular Docker/Kubernetes tools.
The image would automatically bring up the JobManager and wait for TaskManagers to connect to run the embedded application.

Best, Fabian

2018-05-02 1:29 GMT+02:00 anaray <[hidden email]>:

Hi,

I was reading *Docker/Kubernetes* section of FLIP-6
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077).
I have question regarding point 1 ie "A job-specific image with the job
libraries, configured to launch the job (with a JobManager)"

How does job submission happens here ? Do we need write a script to submit
the job?





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[anaray]

Thank You Fabian. We are looking forward for this feature, especially a way
to bundle the job jar along with the container (even in taskmanagers).

In production we deploy flink (1 job=1 cluster) in Docker Swarm. One of the
main issue we face is related to blob download when the taskmanager fails
and restarts . Occasionally we run into blob corruption issue, and job fails
( FLINK-5908 <https://issues.apache.org/jira/browse/FLINK-5908>  ). This
prompted us to monitor job and resubmit the job again using automated
scripts! Since, we go with 1 cluster = 1 job, we can bundle the job jar, and
totally avoid the blob downloads, and automate the deployment & management
of job life-cycle. And we are planning to move to Kubernetes shortly



Best,



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-2]

Hi,

Unfortunately, there won't be native support for Kubernetes in Flink 1.5 yet
[1]. That is, you will still have to deploy a standalone session cluster and
submit a job via the REST API.

Can you open an issue in JIRA regarding the Blob Cache corruption? FLINK-5908 is
closed already.

Best,
Gary

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Migration-to-Flip6-Kubernetes-tp18937p19132.html

On Mon, May 7, 2018 at 7:48 PM, anaray <[hidden email]> wrote:

Thank You Fabian. We are looking forward for this feature, especially a way
to bundle the job jar along with the container (even in taskmanagers).

In production we deploy flink (1 job=1 cluster) in Docker Swarm. One of the
main issue we face is related to blob download when the taskmanager fails
and restarts . Occasionally we run into blob corruption issue, and job fails
( FLINK-5908 <https://issues.apache.org/jira/browse/FLINK-5908>  ). This
prompted us to monitor job and resubmit the job again using automated
scripts! Since, we go with 1 cluster = 1 job, we can bundle the job jar, and
totally avoid the blob downloads, and automate the deployment & management
of job life-cycle. And we are planning to move to Kubernetes shortly



Best,

--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/