Flink Mesos Outstanding Offers - trouble launching task managers

[prashantnayak]

Hi

We’re running Flink 1.3.1 on Mesos.

From time-to-time, the Flink app master seems to have trouble with Mesos offers… At such time, it obviously ends up not launching the requested task managers (mesos.initial-tasks) and we’ve noticed situations where it launches zero tasks.  During such

times we see a long list of “Outstanding Offers” in the Mesos UI.  At the same time, the app master logs have the following

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Processing 12 task(s) against 0 new offer(s) plus outstanding offers.

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Resources considered: (note: expired offers not deducted from below)

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.6 has 0.0 MB, 0.0 cpus

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.233 has 0.0 MB, 0.0 cpus

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Waiting for more offers; 12 task(s) are not yet launched.

The two Mesos agents above (10.80.xx.6, 10.80.xx.233) are listed as having offers outstanding to the Flink framework in the Mesos UI

Appreciate any input on how to go about resolving such an issue.

Thanks

Prashant

[Eron Wright]

Hello, did you resolve this issue?

Thanks,

Eron Wright

On Wed, Jul 12, 2017 at 11:09 AM, Prashant Nayak <[hidden email]> wrote:

Hi

We’re running Flink 1.3.1 on Mesos.

From time-to-time, the Flink app master seems to have trouble with Mesos offers… At such time, it obviously ends up not launching the requested task managers (mesos.initial-tasks) and we’ve noticed situations where it launches zero tasks.  During such

times we see a long list of “Outstanding Offers” in the Mesos UI.  At the same time, the app master logs have the following

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Processing 12 task(s) against 0 new offer(s) plus outstanding offers.

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Resources considered: (note: expired offers not deducted from below)

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.6 has 0.0 MB, 0.0 cpus

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  -   10.80.xx.233 has 0.0 MB, 0.0 cpus

2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.mesos.scheduler.LaunchCoordinator  - Waiting for more offers; 12 task(s) are not yet launched.

The two Mesos agents above (10.80.xx.6, 10.80.xx.233) are listed as having offers outstanding to the Flink framework in the Mesos UI

Appreciate any input on how to go about resolving such an issue.

Thanks

Prashant

[prashantnayak]

Hi Eron

No, unfortunately we did not directly resolve it... we work around it for
now by ensuring that our Mesos slaves are set up to correctly support the
JobManager with offers.

Prashant



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Eron Wright]

Please mail me more information, in particular the JM log and the information on the 'offers' tab on the Mesos UI.   Also, are you using any Mesos roles?

Thanks

On Thu, Aug 31, 2017 at 9:02 AM, prashantnayak <[hidden email]> wrote:

Hi Eron

No, unfortunately we did not directly resolve it... we work around it for
now by ensuring that our Mesos slaves are set up to correctly support the
JobManager with offers.

Prashant



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[prashantnayak]

Thanks… will do this early next week.  Appreciate the follow-up.

Prashant

On Thu, Aug 31, 2017 at 5:45 PM, Eron Wright <[hidden email]> wrote:

Please mail me more information, in particular the JM log and the information on the 'offers' tab on the Mesos UI.   Also, are you using any Mesos roles?

Thanks

On Thu, Aug 31, 2017 at 9:02 AM, prashantnayak <[hidden email]> wrote:

Hi Eron

No, unfortunately we did not directly resolve it... we work around it for
now by ensuring that our Mesos slaves are set up to correctly support the
JobManager with offers.

Prashant



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--