Docker Containers on YARN when using Flink on EMR

[Vijay Balakrishnan]

Hi,

Trying to use Docker Containers to be launched from YARN when using Flink on EMR on Ubuntu. Can't seem to launch a Docker Container from YARN Resource Manager while starting up the ./flink-yarn-session or Submitting a Flink job ./bin/flink run ...

Following the docs here:

https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html#yarn,

Tried various combinations of these variables:

-Dcontainerized.master.env.YARN_CONTAINER_RUNTIME_TYPE=docker,

-Dcontainerized.master.env.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=ashahab/hadoop-trunk

Nothing seems to be working.

TIA,

Vijay