Does flink configuration support configed by environment variables?

[Lifei Chen]

Hi guys,

I am using flink 1.7.2 deployed by kubernetes,  and I want to change the configurations about flink,  for example customize `taskmanager.heap.size`.  

Does flink support using environment variables to override configurations in `conf/flink-conf.yaml` ?

[Stephen Connolly]

I don't think it does. I ended up writing a small CLI tool to enabling templating the file from environment variables. There are loads of such tools, but mine is https://github.com/stephenc/envsub

I have the dockerfile like so:

ARG FLINK_VERSION=1.7.2-alpine

FROM flink:${FLINK_VERSION}

ARG ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c

RUN apk add --no-cache tini curl \

    #

    # Get envsub

    #

\

    && curl -fsSL "<a href="https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub">https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub" -o /usr/local/bin/envsub \

    && if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \

        echo "/usr/local/bin/envsub: Unverified" >&2 ; \

    else \

        echo "${ENVSUB##*::SHA::}  /usr/local/bin/envsub" | sha256sum -c - ; \

    fi \

    && chmod +x /usr/local/bin/envsub 

COPY rootfs/ /

ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]

CMD []

Then docker-entrypoint2.sh looks like

#!/usr/bin/env bash

if [[ "$#" -eq 0 ]]; then

    if [[ -z "${FLINK_ROLE}" ]]; then

        echo "Please set environment variable FLINK_ROLE to either jobmanager or taskmanager"

        exit 1

    fi

    envsub < /opt/flink/conf/flink-conf.template.yaml > /opt/flink/conf/flink-conf.yaml

    exec /docker-entrypoint.sh "${FLINK_ROLE}"

fi

exec /docker-entrypoint.sh "${@}"

and /opt/flink/conf/flink-conf.template.yaml has the environment variable substitution like so:

fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}

fs.s3a.path.style.access: true

fs.s3a.connection.ssl.enabled: false

fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}

fs.s3a.secret.key: ${AWS_SECRET_KEY}

Hope that is sufficient for you to derive your own solution

On Fri, 29 Mar 2019 at 03:09, Lifei Chen <[hidden email]> wrote:

Hi guys,

I am using flink 1.7.2 deployed by kubernetes,  and I want to change the configurations about flink,  for example customize `taskmanager.heap.size`.  

Does flink support using environment variables to override configurations in `conf/flink-conf.yaml` ?

[Lifei Chen]

Thanks,  maybe overwrite configuration in 'conf/flink-conf.yaml' by docker-entrypoint.sh is a common way to do it, thanks for your advice.

Stephen Connolly <[hidden email]> 于2019年4月1日周一 下午10:33写道：

I don't think it does. I ended up writing a small CLI tool to enabling templating the file from environment variables. There are loads of such tools, but mine is https://github.com/stephenc/envsub

I have the dockerfile like so:

ARG FLINK_VERSION=1.7.2-alpine

FROM flink:${FLINK_VERSION}

ARG ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c

RUN apk add --no-cache tini curl \

    #

    # Get envsub

    #

\

    && curl -fsSL "<a href="https://github.com/stephenc/envsub/releases/download/$%7BENVSUB%%::SHA::*%7D/envsub" target="_blank">https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub" -o /usr/local/bin/envsub \

    && if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \

        echo "/usr/local/bin/envsub: Unverified" >&2 ; \

    else \

        echo "${ENVSUB##*::SHA::}  /usr/local/bin/envsub" | sha256sum -c - ; \

    fi \

    && chmod +x /usr/local/bin/envsub 

COPY rootfs/ /

ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]

CMD []

Then docker-entrypoint2.sh looks like

#!/usr/bin/env bash

if [[ "$#" -eq 0 ]]; then

    if [[ -z "${FLINK_ROLE}" ]]; then

        echo "Please set environment variable FLINK_ROLE to either jobmanager or taskmanager"

        exit 1

    fi

    envsub < /opt/flink/conf/flink-conf.template.yaml > /opt/flink/conf/flink-conf.yaml

    exec /docker-entrypoint.sh "${FLINK_ROLE}"

fi

exec /docker-entrypoint.sh "${@}"

and /opt/flink/conf/flink-conf.template.yaml has the environment variable substitution like so:

fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}

fs.s3a.path.style.access: true

fs.s3a.connection.ssl.enabled: false

fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}

fs.s3a.secret.key: ${AWS_SECRET_KEY}

Hope that is sufficient for you to derive your own solution

On Fri, 29 Mar 2019 at 03:09, Lifei Chen <[hidden email]> wrote:

Hi guys,

I am using flink 1.7.2 deployed by kubernetes,  and I want to change the configurations about flink,  for example customize `taskmanager.heap.size`.  

Does flink support using environment variables to override configurations in `conf/flink-conf.yaml` ?