Re: Fwd: Consult about flink on mesos cluster

Posted by Tzu-Li (Gordon) Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Fwd-Consult-about-flink-on-mesos-cluster-tp15982p16058.html

Hi Bo,

I'm not familiar with Mesos deployments, but I'll forward this to Till or Eron (in CC) who perhaps could provide some help here.

Cheers,

Gordon

On 2 October 2017 at 8:49:32 PM, Bo Yu ([hidden email]) wrote:

Hello all,

This is Bo, I met some problems when I tried to use flink in my mesos cluster (1 master, 2 slaves (cpu has 32 cores)).

I tried to start the mesos-appmaster.sh in marathon, the job manager is started without problem.

mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=32

My problem is the task managers are all located in one single slave.

1. (log1)
The initial tasks in "/usr/local/flink/conf/flink-conf.yaml" is setted as "mesos.initial-tasks: 2"

And also set the "mesos.constraints.hard.hostattribute: rack:ak09-27", which is the master node of mesos cluster.

2. (log2)

I tried many ways to distribute the tasks to all the available slaves, and without any success.

So I decide to try add a group_by operator which I referenced from https://mesosphere.github.io/marathon/docs/constraints.html

"mesos.constraints.hard.hostattribute: rack:ak09-27,GROUP_BY:2"

According to the log, flink keep waiting for more offers and the tasks never been launched.

Sorry, I am a newbie to flink, also on mesos. Please reply if my problem is not clear, and I will be appreciate on any hint about how to distribute task evenly on available resources.

Thank you in advance.

Best regards,

Bo