Flink 1.9, MapR secure cluster, high availability

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Stephan Ewen
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Stephan Ewen
Could you share the stack trace where the failure occurs, so we can see why the Flink ZK is used during MapR FS access?

/CC Till and Tison - just FYI

On Fri, Aug 30, 2019 at 9:40 AM Maxim Parkachov <[hidden email]> wrote:
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi Stephan,

sorry for the late answer, didn't have access to cluster.
Here is log and stacktrace.

Hope this helps,
Maxim.

---------------------------------------------------------------------------------------------------------------------------------
2019-09-16 18:00:31,804 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-16 18:00:31,806 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting YarnSessionClusterEntrypoint (Version: 1.9.0, Rev:9c32ed9, Date:19.08.2019 @ 16:16:55 UTC)
2019-09-16 18:00:31,806 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: run
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: run
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.152-b16
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 161 MiBytes
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /pb/apps/java/jdk1.8.0_152/
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Hadoop version: 2.7.0-mapr-1808
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms168m
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx168m
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.log
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:log4j.properties
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments: (none)
2019-09-16 18:00:32,287 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-16 18:00:32,287 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-16 18:00:32,288 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-09-16 18:00:32,290 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - YARN daemon is running as: run Yarn client user obtainer: run
2019-09-16 18:00:32,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: fs.default-scheme, maprfs:///user/run/flink/
2019-09-16 18:00:32,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.tags, flinksessioncluster
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.cluster-id, application_1568125115996_13508
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: FLINK_PLUGINS_DIR, /home/run/flink-1.9.0/plugins
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.savepoints.dir, maprfs:///user/run/flink/savepoints/
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.path.root, /flink
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.storageDir, maprfs:///user/run/flink/recovery
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.quorum, server11.internal.com:5181
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: internal.cluster.execution-mode, NORMAL
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability, zookeeper
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.application-attempts, 10
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend, filesystem
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 768m
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.checkpoints.dir, maprfs:///user/run/flink/checkpoints/
2019-09-16 18:00:32,294 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 768m
2019-09-16 18:00:32,311 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-16 18:00:32,314 INFO  org.apache.flink.runtime.clusterframework.BootstrapTools      - Setting directories for temporary files to: /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508
2019-09-16 18:00:32,318 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting YarnSessionClusterEntrypoint.
2019-09-16 18:00:32,318 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
2019-09-16 18:00:32,318 WARN  org.apache.flink.core.plugin.PluginConfig                     - Environment variable [FLINK_PLUGINS_DIR] is set to [/home/run/flink-1.9.0/plugins] but the directory doesn't exist
2019-09-16 18:00:32,359 WARN  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop security is enabled but current login user does not have Kerberos credentials
2019-09-16 18:00:32,359 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to run (auth:CUSTOM)
2019-09-16 18:00:32,362 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
2019-09-16 18:00:32,475 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-16 18:00:32,802 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-16 18:00:32,820 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-16 18:00:32,911 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@...:45399]
2019-09-16 18:00:33,006 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@...:45399
2019-09-16 18:00:33,018 INFO  org.apache.flink.runtime.fs.maprfs.MapRFsFactory              - Trying to load and instantiate MapR File System
2019-09-16 18:00:33,062 INFO  org.apache.flink.runtime.blob.FileSystemBlobStore             - Creating highly available BLOB storage directory at maprfs:///user/run/flink/recovery/application_1568125115996_13508/blob
2019-09-16 18:00:33,091 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Enforcing default ACL for ZK connections
2019-09-16 18:00:33,092 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink/application_1568125115996_13508' as Zookeeper namespace.
2019-09-16 18:00:33,132 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:host.name=server11.internal.com
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_152
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.home=/pb/apps/java/jdk1.8.0_152/jre
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=/tmp
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Linux
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.version=3.10.0-957.21.3.el7.x86_64
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.name=run
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.home=/home/run
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=/tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=server11.internal.com:5181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@6b530eb9
2019-09-16 18:00:33,151 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login  - Client successfully logged in.
2019-09-16 18:00:33,155 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - Client will use GSSAPI as SASL mechanism.
2019-09-16 18:00:33,155 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/blobStore-59e8dea5-cabf-441e-8fbd-598396a22f27
2019-09-16 18:00:33,158 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:33509 - max concurrent requests: 50 - max backlog: 1000
2019-09-16 18:00:33,160 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server server11.internal.com/10.200.133.203:5181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-09-16 18:00:33,162 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to server11.internal.com/10.200.133.203:5181, initiating session
2019-09-16 18:00:33,167 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server server11.internal.com/10.200.133.203:5181, sessionid = 0x100af52c3c16c26, negotiated timeout = 40000
2019-09-16 18:00:33,169 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
2019-09-16 18:00:33,170 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-09-16 18:00:33,170 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-09-16 18:00:33,170 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-16 18:00:33,172 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2019-09-16 18:00:33,172 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-09-16 18:00:33,184 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-16 18:00:33,186 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-16 18:00:33,199 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@...:36539]
2019-09-16 18:00:33,227 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink-metrics@...:36539
2019-09-16 18:00:33,232 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService .
2019-09-16 18:00:33,240 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/executionGraphStore-0af12230-9fff-4481-ba1c-13841eb6a679, expiration time 3600000, maximum cache size 52428800 bytes.
2019-09-16 18:00:33,277 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-16 18:00:33,279 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-ecf8f921-2fa6-4e2a-94c4-302862c79f7b/flink-web-upload does not exist.
2019-09-16 18:00:33,279 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-ecf8f921-2fa6-4e2a-94c4-302862c79f7b/flink-web-upload for file uploads.
2019-09-16 18:00:33,280 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.
2019-09-16 18:00:33,445 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.log
2019-09-16 18:00:33,445 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.out
2019-09-16 18:00:33,600 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at server11.internal.com:40850
2019-09-16 18:00:33,601 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2019-09-16 18:00:48,614 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Connection timed out for connection string (server11.internal.com:5181) and timeout (15000) / elapsed (15440)
org.apache.flink.shaded.curator.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService.start(ZooKeeperLeaderElectionService.java:134)
at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.startInternal(WebMonitorEndpoint.java:666)
at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:241)
at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:161)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)
at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93)
2019-09-16 18:00:48,619 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Connection timed out for connection string (server11.internal.com:5181) and timeout (15000) / elapsed (15447)
org.apache.flink.shaded.curator.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:599)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:465)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.reset(LeaderLatch.java:505)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.internalStart(LeaderLatch.java:515)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.access$100(LeaderLatch.java:64)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$3.run(LeaderLatch.java:169)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.AfterConnectionEstablished$1.run(AfterConnectionEstablished.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

On Fri, Aug 30, 2019 at 2:41 PM Stephan Ewen <[hidden email]> wrote:
Could you share the stack trace where the failure occurs, so we can see why the Flink ZK is used during MapR FS access?

/CC Till and Tison - just FYI

On Fri, Aug 30, 2019 at 9:40 AM Maxim Parkachov <[hidden email]> wrote:
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Stephan Ewen
Hi!

Not sure what is happening here. 

  - I cannot understand why MapR FS should use Flink's relocated ZK dependency
  - It might be that it doesn't and that all the logging we see probably comes from Flink's HA services. Maybe the MapR stuff uses a different logging framework and the logs do not get forwarded (there is suspiciously no MapR related log line at all in the code)
  - Then it might be that Flink is simply not set up with the correct credentials to work against ZK.

Best,
Stephan


On Mon, Sep 16, 2019 at 6:14 PM Maxim Parkachov <[hidden email]> wrote:
Hi Stephan,

sorry for the late answer, didn't have access to cluster.
Here is log and stacktrace.

Hope this helps,
Maxim.

---------------------------------------------------------------------------------------------------------------------------------
2019-09-16 18:00:31,804 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-16 18:00:31,806 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting YarnSessionClusterEntrypoint (Version: 1.9.0, Rev:9c32ed9, Date:19.08.2019 @ 16:16:55 UTC)
2019-09-16 18:00:31,806 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: run
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: run
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.152-b16
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 161 MiBytes
2019-09-16 18:00:32,285 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /pb/apps/java/jdk1.8.0_152/
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Hadoop version: 2.7.0-mapr-1808
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms168m
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx168m
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.log
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:log4j.properties
2019-09-16 18:00:32,286 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments: (none)
2019-09-16 18:00:32,287 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-16 18:00:32,287 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-16 18:00:32,288 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-09-16 18:00:32,290 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - YARN daemon is running as: run Yarn client user obtainer: run
2019-09-16 18:00:32,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: fs.default-scheme, maprfs:///user/run/flink/
2019-09-16 18:00:32,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.tags, flinksessioncluster
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.cluster-id, application_1568125115996_13508
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: FLINK_PLUGINS_DIR, /home/run/flink-1.9.0/plugins
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.savepoints.dir, maprfs:///user/run/flink/savepoints/
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.path.root, /flink
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.storageDir, maprfs:///user/run/flink/recovery
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.quorum, server11.internal.com:5181
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: internal.cluster.execution-mode, NORMAL
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability, zookeeper
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.application-attempts, 10
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend, filesystem
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 768m
2019-09-16 18:00:32,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.checkpoints.dir, maprfs:///user/run/flink/checkpoints/
2019-09-16 18:00:32,294 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 768m
2019-09-16 18:00:32,311 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-16 18:00:32,314 INFO  org.apache.flink.runtime.clusterframework.BootstrapTools      - Setting directories for temporary files to: /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508
2019-09-16 18:00:32,318 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting YarnSessionClusterEntrypoint.
2019-09-16 18:00:32,318 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
2019-09-16 18:00:32,318 WARN  org.apache.flink.core.plugin.PluginConfig                     - Environment variable [FLINK_PLUGINS_DIR] is set to [/home/run/flink-1.9.0/plugins] but the directory doesn't exist
2019-09-16 18:00:32,359 WARN  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop security is enabled but current login user does not have Kerberos credentials
2019-09-16 18:00:32,359 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to run (auth:CUSTOM)
2019-09-16 18:00:32,362 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
2019-09-16 18:00:32,475 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-16 18:00:32,802 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-16 18:00:32,820 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-16 18:00:32,911 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@...:45399]
2019-09-16 18:00:33,006 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@...:45399
2019-09-16 18:00:33,018 INFO  org.apache.flink.runtime.fs.maprfs.MapRFsFactory              - Trying to load and instantiate MapR File System
2019-09-16 18:00:33,062 INFO  org.apache.flink.runtime.blob.FileSystemBlobStore             - Creating highly available BLOB storage directory at maprfs:///user/run/flink/recovery/application_1568125115996_13508/blob
2019-09-16 18:00:33,091 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Enforcing default ACL for ZK connections
2019-09-16 18:00:33,092 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink/application_1568125115996_13508' as Zookeeper namespace.
2019-09-16 18:00:33,132 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:host.name=server11.internal.com
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_152
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.home=/pb/apps/java/jdk1.8.0_152/jre
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=/tmp
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Linux
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.version=3.10.0-957.21.3.el7.x86_64
2019-09-16 18:00:33,138 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.name=run
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.home=/home/run
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=/tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001
2019-09-16 18:00:33,139 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=server11.internal.com:5181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@6b530eb9
2019-09-16 18:00:33,151 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login  - Client successfully logged in.
2019-09-16 18:00:33,155 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - Client will use GSSAPI as SASL mechanism.
2019-09-16 18:00:33,155 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/blobStore-59e8dea5-cabf-441e-8fbd-598396a22f27
2019-09-16 18:00:33,158 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:33509 - max concurrent requests: 50 - max backlog: 1000
2019-09-16 18:00:33,160 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server server11.internal.com/10.200.133.203:5181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-09-16 18:00:33,162 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to server11.internal.com/10.200.133.203:5181, initiating session
2019-09-16 18:00:33,167 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server server11.internal.com/10.200.133.203:5181, sessionid = 0x100af52c3c16c26, negotiated timeout = 40000
2019-09-16 18:00:33,169 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
2019-09-16 18:00:33,170 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-09-16 18:00:33,170 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-09-16 18:00:33,170 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-16 18:00:33,172 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2019-09-16 18:00:33,172 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-09-16 18:00:33,184 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-16 18:00:33,186 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-16 18:00:33,199 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@...:36539]
2019-09-16 18:00:33,227 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink-metrics@...:36539
2019-09-16 18:00:33,232 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService .
2019-09-16 18:00:33,240 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_13508/executionGraphStore-0af12230-9fff-4481-ba1c-13841eb6a679, expiration time 3600000, maximum cache size 52428800 bytes.
2019-09-16 18:00:33,277 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-16 18:00:33,279 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-ecf8f921-2fa6-4e2a-94c4-302862c79f7b/flink-web-upload does not exist.
2019-09-16 18:00:33,279 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-ecf8f921-2fa6-4e2a-94c4-302862c79f7b/flink-web-upload for file uploads.
2019-09-16 18:00:33,280 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.
2019-09-16 18:00:33,445 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.log
2019-09-16 18:00:33,445 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_13508/container_e64_1568125115996_13508_01_000001/jobmanager.out
2019-09-16 18:00:33,600 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at server11.internal.com:40850
2019-09-16 18:00:33,601 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2019-09-16 18:00:48,614 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Connection timed out for connection string (server11.internal.com:5181) and timeout (15000) / elapsed (15440)
org.apache.flink.shaded.curator.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService.start(ZooKeeperLeaderElectionService.java:134)
at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.startInternal(WebMonitorEndpoint.java:666)
at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:241)
at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:161)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)
at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93)
2019-09-16 18:00:48,619 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  - Connection timed out for connection string (server11.internal.com:5181) and timeout (15000) / elapsed (15447)
org.apache.flink.shaded.curator.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.curator.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:599)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:465)
at org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.reset(LeaderLatch.java:505)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.internalStart(LeaderLatch.java:515)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.access$100(LeaderLatch.java:64)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$3.run(LeaderLatch.java:169)
at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.AfterConnectionEstablished$1.run(AfterConnectionEstablished.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

On Fri, Aug 30, 2019 at 2:41 PM Stephan Ewen <[hidden email]> wrote:
Could you share the stack trace where the failure occurs, so we can see why the Flink ZK is used during MapR FS access?

/CC Till and Tison - just FYI

On Fri, Aug 30, 2019 at 9:40 AM Maxim Parkachov <[hidden email]> wrote:
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi Stephan,

I have compiled flink with MapR dependencies, including zookeeper. It is still uses shaded zookeeper, but now it is mapr zookeeper. I have changed nothing else and everything works. Bellow you could find log of successful start with compiled flink.


Not sure what is happening here. 

  - I cannot understand why MapR FS should use Flink's relocated ZK dependency
  - It might be that it doesn't and that all the logging we see probably comes from Flink's HA services. Maybe the MapR stuff uses a different logging framework and the logs do not get forwarded (there is suspiciously no MapR related log line at all in the code)
  - Then it might be that Flink is simply not set up with the correct credentials to work against ZK.

I did checked all parameters, everything seems to be correctly setup.
As far as I know, the vanilla zookeeper client could not connect to secured mapr zookeeper, which is probably why it doesn't work. The only question as you mentioned above, why it is using shaded zookeeper, when there is mapr zookeeper in classpath.

Hope this helps,
Maxim.
-------------------------------------------------------------------
2019-09-20 11:45:25,713 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-20 11:45:25,714 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting YarnSessionClusterEntrypoint (Version: 1.9.0, Rev:<unknown>, Date:<unknown>)
2019-09-20 11:45:25,714 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: run
2019-09-20 11:45:26,205 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: run
2019-09-20 11:45:26,205 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.152-b16
2019-09-20 11:45:26,205 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 161 MiBytes
2019-09-20 11:45:26,205 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /pb/apps/java/jdk1.8.0_152/
2019-09-20 11:45:26,206 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Hadoop version: 2.7.0-mapr-1808
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms168m
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx168m
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_24664/container_e64_1568125115996_24664_01_000001/jobmanager.log
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:log4j.properties
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments: (none)
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-20 11:45:26,207 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-09-20 11:45:26,208 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-09-20 11:45:26,210 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - YARN daemon is running as: run Yarn client user obtainer: run
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: fs.default-scheme, maprfs:///user/run/flink/
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.tags, flinksessioncluster
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.cluster-id, application_1568125115996_24664
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: FLINK_PLUGINS_DIR, /home/run/flink-1.9.0/plugins
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.savepoints.dir, maprfs:///user/run/flink/savepoints/
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.path.root, /flink
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.storageDir, maprfs:///user/run/flink/recovery
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.quorum, server11.internal.com:5181
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: internal.cluster.execution-mode, NORMAL
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability, zookeeper
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.application-attempts, 10
2019-09-20 11:45:26,213 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-09-20 11:45:26,214 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend, filesystem
2019-09-20 11:45:26,214 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 768m
2019-09-20 11:45:26,214 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.checkpoints.dir, maprfs:///user/run/flink/checkpoints/
2019-09-20 11:45:26,214 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 768m
2019-09-20 11:45:26,248 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-20 11:45:26,251 INFO  org.apache.flink.runtime.clusterframework.BootstrapTools      - Setting directories for temporary files to: /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_24664
2019-09-20 11:45:26,257 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting YarnSessionClusterEntrypoint.
2019-09-20 11:45:26,257 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
2019-09-20 11:45:26,258 WARN  org.apache.flink.core.plugin.PluginConfig                     - Environment variable [FLINK_PLUGINS_DIR] is set to [/home/run/flink-1.9.0/plugins] but the directory doesn't exist
2019-09-20 11:45:26,312 WARN  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop security is enabled but current login user does not have Kerberos credentials
2019-09-20 11:45:26,312 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to run (auth:CUSTOM)
2019-09-20 11:45:26,315 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
2019-09-20 11:45:26,427 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-20 11:45:26,784 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-20 11:45:26,811 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-20 11:45:26,897 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@...:42158]
2019-09-20 11:45:27,020 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@...:42158
2019-09-20 11:45:27,033 INFO  org.apache.flink.runtime.fs.maprfs.MapRFsFactory              - Trying to load and instantiate MapR File System
2019-09-20 11:45:27,086 INFO  org.apache.flink.runtime.blob.FileSystemBlobStore             - Creating highly available BLOB storage directory at maprfs:///user/run/flink/recovery/application_1568125115996_24664/blob
2019-09-20 11:45:27,117 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Enforcing default ACL for ZK connections
2019-09-20 11:45:27,118 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink/application_1568125115996_24664' as Zookeeper namespace.
2019-09-20 11:45:27,157 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2019-09-20 11:45:27,162 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:zookeeper.version=3.4.11-mapr-1808--1, built on 09/27/2018 07:40 GMT
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:host.name=server11.internal.com
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.version=1.8.0_152
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.vendor=Oracle Corporation
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.home=/pb/apps/java/jdk1.8.0_152/jre
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.class.path=log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.io.tmpdir=/tmp
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:java.compiler=<NA>
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.name=Linux
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.arch=amd64
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:os.version=3.10.0-957.21.3.el7.x86_64
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.name=run
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.home=/home/run
2019-09-20 11:45:27,163 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Client environment:user.dir=/tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_24664/container_e64_1568125115996_24664_01_000001
2019-09-20 11:45:27,164 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  - Initiating client connection, connectString=server11.internal.com:5181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@5ab9b447
2019-09-20 11:45:27,177 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login  - Client successfully logged in.
2019-09-20 11:45:27,179 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - Client will use MAPR-SECURITY as SASL mechanism.
2019-09-20 11:45:27,181 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server server11.internal.com/10.200.133.203:5181. Will attempt to SASL-authenticate using Login Context section 'Client', mechanism MAPR-SECURITY, principal zookeeper/server11.internal.com
2019-09-20 11:45:27,184 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to server11.internal.com/10.200.133.203:5181, initiating session
2019-09-20 11:45:27,190 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_24664/blobStore-d188804a-0c43-4dfd-85a2-d85eb4d57503
2019-09-20 11:45:27,195 INFO  org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server server11.internal.com/10.200.133.203:5181, sessionid = 0x100af52c3c30037, negotiated timeout = 40000
2019-09-20 11:45:27,213 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:34528 - max concurrent requests: 50 - max backlog: 1000
2019-09-20 11:45:27,216 INFO  org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2019-09-20 11:45:27,235 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
2019-09-20 11:45:27,238 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at server11.internal.com:0
2019-09-20 11:45:27,264 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-09-20 11:45:27,268 INFO  akka.remote.Remoting                                          - Starting remoting
2019-09-20 11:45:27,287 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@...:34525]
2019-09-20 11:45:27,322 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink-metrics@...:34525
2019-09-20 11:45:27,330 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService .
2019-09-20 11:45:27,343 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_24664/executionGraphStore-be3b9d5b-7262-475c-a3da-b73fed6d8bf1, expiration time 3600000, maximum cache size 52428800 bytes.
2019-09-20 11:45:27,384 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-09-20 11:45:27,385 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-2f54956a-ef91-4f95-9e6a-db8ca95531bb/flink-web-upload does not exist.
2019-09-20 11:45:27,386 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-2f54956a-ef91-4f95-9e6a-db8ca95531bb/flink-web-upload for file uploads.
2019-09-20 11:45:27,387 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.
2019-09-20 11:45:27,574 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_24664/container_e64_1568125115996_24664_01_000001/jobmanager.log
2019-09-20 11:45:27,575 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1568125115996_24664/container_e64_1568125115996_24664_01_000001/jobmanager.out
2019-09-20 11:45:27,749 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at server11.internal.com:45846
2019-09-20 11:45:27,751 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}.
2019-09-20 11:45:27,766 WARN  org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths  - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
2019-09-20 11:45:27,776 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://server11.internal.com:45846.
2019-09-20 11:45:27,817 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - http://server11.internal.com:45846 was granted leadership with leaderSessionID=0aeedcfe-ddaa-4994-84ea-6471ee2e5d16
2019-09-20 11:45:27,849 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
2019-09-20 11:45:27,895 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2019-09-20 11:45:27,904 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-09-20 11:45:27,905 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock.
2019-09-20 11:45:27,908 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}.
2019-09-20 11:45:27,927 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@...:42158/user/dispatcher was granted leadership with fencing token 59f8714d-c951-44d0-a7cb-0dea93c7883d
2019-09-20 11:45:27,929 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2019-09-20 11:45:27,972 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:zookeeper.version=3.4.11-mapr-1808--1, built on 09/27/2018 07:40 GMT
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:host.name=server11.internal.com
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.version=1.8.0_152
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.vendor=Oracle Corporation
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.home=/pb/apps/java/jdk1.8.0_152/jre
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.class.path=log4j.properties:flink.jar:flink-conf.yaml::/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-spark-yarn-shuffle.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-smart-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-data-lake-store-sdk-2.1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.54.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/antlr4-runtime-4.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/aws-java-sdk-bundle-1.11.199.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/azure-storage-2.0.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-java-utils-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-scala-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-tools-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-aws-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-azure-datalake-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-cdc-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-mapreduce-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprdb-shell-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-streams-6.1.0-mapr-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/ojai-mapreduce-3.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-client-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-api-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guava-14.0.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/eval-0.5-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808-tests.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/zookeeper-3.4.11-mapr-1808.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/audience-annotations-0.5.0.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/objenesis-2.1.jar
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.io.tmpdir=/tmp
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.compiler=<NA>
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.name=Linux
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.arch=amd64
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.version=3.10.0-957.21.3.el7.x86_64
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.name=run
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.home=/home/run
2019-09-20 11:45:27,973 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.dir=/tmp/hadoop-mapr/nm-local-dir/usercache/run/appcache/application_1568125115996_24664/container_e64_1568125115996_24664_01_000001
2019-09-20 11:45:27,974 INFO  org.apache.zookeeper.ZooKeeper                                - Initiating client connection, connectString=server11.internal.com:5181 sessionTimeout=30000 watcher=com.mapr.util.zookeeper.ZKDataRetrieval@76443300
2019-09-20 11:45:27,985 INFO  org.apache.zookeeper.Login                                    - Client successfully logged in.
2019-09-20 11:45:27,986 INFO  org.apache.zookeeper.client.ZooKeeperSaslClient               - Client will use MAPR-SECURITY as SASL mechanism.
2019-09-20 11:45:27,986 INFO  org.apache.zookeeper.ClientCnxn                               - Opening socket connection to server server11.internal.com/10.200.133.203:5181. Will attempt to SASL-authenticate using Login Context section 'Client', mechanism MAPR-SECURITY, principal zookeeper/server11.internal.com
2019-09-20 11:45:27,986 INFO  org.apache.zookeeper.ClientCnxn                               - Socket connection established to server11.internal.com/10.200.133.203:5181, initiating session
2019-09-20 11:45:27,992 INFO  org.apache.zookeeper.ClientCnxn                               - Session establishment complete on server server11.internal.com/10.200.133.203:5181, sessionid = 0x100af52c3c30038, negotiated timeout = 30000
2019-09-20 11:45:27,993 INFO  com.mapr.util.zookeeper.ZKDataRetrieval                       - Process path: null. Event state: SyncConnected. Event type: None
2019-09-20 11:45:27,993 INFO  com.mapr.util.zookeeper.ZKDataRetrieval                       - Connected to ZK: server11.internal.com:5181
2019-09-20 11:45:27,994 INFO  com.mapr.util.zookeeper.ZKDataRetrieval                       - Getting serviceData for master node of resourcemanager
2019-09-20 11:45:27,995 INFO  com.mapr.util.zookeeper.ZKDataRetrieval                       - Process path: null. Event state: SaslAuthenticated. Event type: None
2019-09-20 11:45:28,006 INFO  org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider  - Updated RM address to server11.internal.com/10.200.133.203:8030
2019-09-20 11:45:28,210 INFO  org.apache.flink.yarn.YarnResourceManager                     - Recovered 0 containers from previous attempts ([]).
2019-09-20 11:45:28,212 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - yarn.client.max-cached-nodemanagers-proxies : 0
2019-09-20 11:45:28,212 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
2019-09-20 11:45:28,229 INFO  org.apache.flink.yarn.YarnResourceManager                     - ResourceManager akka.tcp://flink@...:42158/user/resourcemanager was granted leadership with fencing token 9032c6a4b9364a55e2911c8625b14ba2
2019-09-20 11:45:28,231 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl  - Starting the SlotManager.