(DEPRECATED) Apache Flink User Mailing List archive.

Container running beyond physical memory limits when processing DataStream

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
Jack Huang-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Container running beyond physical memory limits when processing DataStream

Jack Huang-2
Hi all,

I am running a test Flink streaming task under YARN. It reads messages from a Kafka topic and writes them to local file system. 
object PricerEvent {
    def main(args:Array[String]) {
        val kafkaProp = new Properties()
        kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
        kafkaProp.setProperty("auto.offset.reset", "earliest")

        val env = StreamExecutionEnvironment.getExecutionEnvironment
        env.setStateBackend(new MemoryStateBackend)

        val wins = env.addSource(new FlinkKafkaConsumer09[String]("events", new SimpleStringSchema, kafkaProp))
        wins.writeAsText("/home/user/flink_out/" + new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))

        env.execute
    }
}
​
With the command
flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent /home/user/flink-example/build/libs/flink-example-1.0-all.jar
​

The task runs fine for a moment and then terminates. I looked into the error log and found following out-of-memory error message:
2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager                          - Container container_e05_1467433388200_0136_01_000002 is completed with diagnostics: Container [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779 /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m -XX:MaxDirectMemorySize=1448m -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 
    |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m -XX:MaxDirectMemorySize=1448m  -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out 2> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
​

I don't understand how it should fail due to out-of-memory error. I would expect the task to slow down if there are too many messages to process, but not failing altogether. I am not storing any states either. 

Does anyone know the reason and the way to fix/avoid this issue?


Thanks,
Jack
Maximilian Michels
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Maximilian Michels
Hi Jack,

Considering the type of job you're running, you shouldn't run out of
memory. Could it be that the events are quite large strings? It could
be that the TextOutputFormat doesn't write to disk fast enough and
accumulates memory. Actually, it doesn't perform regular flushing
which could be an issue.

I'm just guessing, we need to investigate further. Could you please
supply the entire JobManager log file output?

Thanks,
Max

On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:

> Hi all,
>
> I am running a test Flink streaming task under YARN. It reads messages from
> a Kafka topic and writes them to local file system.
>
> object PricerEvent {
>     def main(args:Array[String]) {
>         val kafkaProp = new Properties()
>         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>         kafkaProp.setProperty("auto.offset.reset", "earliest")
>
>         val env = StreamExecutionEnvironment.getExecutionEnvironment
>         env.setStateBackend(new MemoryStateBackend)
>
>         val wins = env.addSource(new FlinkKafkaConsumer09[String]("events",
> new SimpleStringSchema, kafkaProp))
>         wins.writeAsText("/home/user/flink_out/" + new
> SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>
>         env.execute
>     }
> }
>
> With the command
>
> flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
> /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>
>
> The task runs fine for a moment and then terminates. I looked into the error
> log and found following out-of-memory error message:
>
> 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
> - Container container_e05_1467433388200_0136_01_000002 is completed with
> diagnostics: Container
> [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running
> beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory
> used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
> 2>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> I don't understand how it should fail due to out-of-memory error. I would
> expect the task to slow down if there are too many messages to process, but
> not failing altogether. I am not storing any states either.
>
> Does anyone know the reason and the way to fix/avoid this issue?
>
>
> Thanks,
> Jack
>
>
>
>
Jack Huang-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Jack Huang-2
Hi Max,

Each events are only a few hundred bytes. I am reading from a Kafka topic from some offset in the past, so the events should be flowing in as fast as Flink can process them.

The entire YARN task log, which contains both JobManager and TaskManager outputs, is attached.

Thanks a lot,
Jack


On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]> wrote:
Hi Jack,

Considering the type of job you're running, you shouldn't run out of
memory. Could it be that the events are quite large strings? It could
be that the TextOutputFormat doesn't write to disk fast enough and
accumulates memory. Actually, it doesn't perform regular flushing
which could be an issue.

I'm just guessing, we need to investigate further. Could you please
supply the entire JobManager log file output?

Thanks,
Max

On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
> Hi all,
>
> I am running a test Flink streaming task under YARN. It reads messages from
> a Kafka topic and writes them to local file system.
>
> object PricerEvent {
>     def main(args:Array[String]) {
>         val kafkaProp = new Properties()
>         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>         kafkaProp.setProperty("auto.offset.reset", "earliest")
>
>         val env = StreamExecutionEnvironment.getExecutionEnvironment
>         env.setStateBackend(new MemoryStateBackend)
>
>         val wins = env.addSource(new FlinkKafkaConsumer09[String]("events",
> new SimpleStringSchema, kafkaProp))
>         wins.writeAsText("/home/user/flink_out/" + new
> SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>
>         env.execute
>     }
> }
>
> With the command
>
> flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
> /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>
>
> The task runs fine for a moment and then terminates. I looked into the error
> log and found following out-of-memory error message:
>
> 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
> - Container container_e05_1467433388200_0136_01_000002 is completed with
> diagnostics: Container
> [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running
> beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory
> used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
> 2>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> I don't understand how it should fail due to out-of-memory error. I would
> expect the task to slow down if there are too many messages to process, but
> not failing altogether. I am not storing any states either.
>
> Does anyone know the reason and the way to fix/avoid this issue?
>
>
> Thanks,
> Jack
>
>
>
>


flink.log (139K) Download Attachment
Jack Huang-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Jack Huang-2
Hi Max, here is the YARN task log. Searching for "limit" will bring you to the error line.

Container: container_e05_1467433388200_0136_01_000002 on node03.mycluster.com_45454
============================================================================================
LogType:taskmanager.err
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:0
Log Contents:
End of LogType:taskmanager.err

LogType:taskmanager.log
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:49515
Log Contents:
2016-07-28 22:34:12,861 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - --------------------------------------------------------------------------------
2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Starting YARN TaskManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC)
2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Current user: yarn
2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.60-b23
2016-07-28 22:34:12,872 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Maximum heap size: 1388 MiBytes
2016-07-28 22:34:12,872 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  JAVA_HOME: /usr/jdk64/jdk1.8.0_60
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Hadoop version: 2.7.2
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  JVM Options:
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -Xms1448m
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -Xmx1448m
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -XX:MaxDirectMemorySize=1448m
2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -Dlogback.configurationFile=file:logback.xml
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     -Dlog4j.configuration=file:log4j.properties
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Program Arguments:
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     --configDir
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -     .
2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   -  Classpath: /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar
2016-07-28 22:34:12,877 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - --------------------------------------------------------------------------------
2016-07-28 22:34:12,878 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - Registered UNIX signal handlers for [TERM, HUP, INT]
2016-07-28 22:34:13,098 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Loading configuration from .
2016-07-28 22:34:13,111 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - Setting directories for temporary file /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136
2016-07-28 22:34:13,111 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - YARN daemon runs as 'yarn' setting user to execute Flink TaskManager to 'user'
2016-07-28 22:34:13,126 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink' as root namespace.
2016-07-28 22:34:13,224 INFO  org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:host.name=node03.mycluster.com
2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.version=1.8.0_60
2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.vendor=Oracle Corporation
2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
2016-07-28 22:34:13,243 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.class.path=/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.library.path=::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.io.tmpdir=/tmp
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.compiler=<NA>
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.name=Linux
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.arch=amd64
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.version=3.13.0-32-generic
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.name=yarn
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.home=/home/yarn
2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.dir=/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002
2016-07-28 22:34:13,245 INFO  org.apache.zookeeper.ZooKeeper                                - Initiating client connection, connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@5f8e8a9d
2016-07-28 22:34:13,314 INFO  org.apache.zookeeper.ClientCnxn                               - Opening socket connection to server node03.mycluster.com/38.111.30.146:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 22:34:13,327 INFO  org.apache.zookeeper.ClientCnxn                               - Socket connection established to node03.mycluster.com/38.111.30.146:2181, initiating session
2016-07-28 22:34:13,354 INFO  org.apache.zookeeper.ClientCnxn                               - Session establishment complete on server node03.mycluster.com/38.111.30.146:2181, sessionid = 0x3556f328b260300, negotiated timeout = 40000
2016-07-28 22:34:13,360 INFO  org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2016-07-28 22:34:14,430 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService.
2016-07-28 22:34:14,436 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to select the network interface and address to use by connecting to the leading JobManager.
2016-07-28 22:34:14,437 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
2016-07-28 22:34:14,549 INFO  org.apache.flink.runtime.net.ConnectionUtils                  - Retrieved new target address /38.111.30.148:59857.
2016-07-28 22:34:14,578 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping ZooKeeperLeaderRetrievalService.
2016-07-28 22:34:14,590 INFO  org.apache.zookeeper.ZooKeeper                                - Session: 0x3556f328b260300 closed
2016-07-28 22:34:14,590 INFO  org.apache.zookeeper.ClientCnxn                               - EventThread shut down
2016-07-28 22:34:14,597 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager will use hostname/address 'node03' (38.111.30.146) for communication.
2016-07-28 22:34:14,598 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager
2016-07-28 22:34:14,599 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor system at 38.111.30.146:0
2016-07-28 22:34:15,001 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2016-07-28 22:34:15,046 INFO  Remoting                                                      - Starting remoting
2016-07-28 22:34:15,353 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@38.111.30.146:48700]
2016-07-28 22:34:15,360 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor
2016-07-28 22:34:15,369 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig [server address: node03/38.111.30.146, server port: 40641, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 1 (manual), number of client threads: 1 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2016-07-28 22:34:15,372 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Messages between TaskManager and JobManager have a max timeout of 10000 milliseconds
2016-07-28 22:34:15,378 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136': total 2734 GB, usable 2055 GB (75.16% usable)
2016-07-28 22:34:15,379 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136': total 2750 GB, usable 2064 GB (75.05% usable)
2016-07-28 22:34:15,379 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136': total 2750 GB, usable 2233 GB (81.20% usable)
2016-07-28 22:34:15,380 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136': total 2750 GB, usable 2466 GB (89.67% usable)
2016-07-28 22:34:15,416 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
2016-07-28 22:34:15,599 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Limiting managed memory to 0.7 of the currently free heap space (922 MB), memory will be allocated lazily.
2016-07-28 22:34:15,626 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-96e84170-f59d-4316-bda1-d2c2810fdcdf for spill files.
2016-07-28 22:34:15,626 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-996962ff-d621-4285-a4d2-7eb9e846ee11 for spill files.
2016-07-28 22:34:15,626 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-4b2cc5e6-110c-43aa-a1b1-620058e12a94 for spill files.
2016-07-28 22:34:15,627 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-a9c9d249-da5c-4c3c-a208-06603df1e966 for spill files.
2016-07-28 22:34:15,630 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink' as root namespace.
2016-07-28 22:34:15,630 INFO  org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2016-07-28 22:34:15,631 INFO  org.apache.zookeeper.ZooKeeper                                - Initiating client connection, connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@4678a2eb
2016-07-28 22:34:15,634 INFO  org.apache.zookeeper.ClientCnxn                               - Opening socket connection to server node02.mycluster.com/38.111.30.145:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 22:34:15,634 INFO  org.apache.zookeeper.ClientCnxn                               - Socket connection established to node02.mycluster.com/38.111.30.145:2181, initiating session
2016-07-28 22:34:15,656 INFO  org.apache.zookeeper.ClientCnxn                               - Session establishment complete on server node02.mycluster.com/38.111.30.145:2181, sessionid = 0x255bc6698ff025f, negotiated timeout = 40000
2016-07-28 22:34:15,656 INFO  org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
2016-07-28 22:34:15,964 INFO  org.apache.flink.yarn.YarnTaskManager                         - Starting TaskManager actor at akka://flink/user/taskmanager#-1203652007.
2016-07-28 22:34:15,964 INFO  org.apache.flink.yarn.YarnTaskManager                         - TaskManager data connection information: node03.mycluster.com (dataPort=40641)
2016-07-28 22:34:15,965 INFO  org.apache.flink.yarn.YarnTaskManager                         - TaskManager has 1 task slot(s).
2016-07-28 22:34:15,967 INFO  org.apache.flink.yarn.YarnTaskManager                         - Memory usage stats: [HEAP: 123/1388/1388 MB, NON HEAP: 33/34/-1 MB (used/committed/max)]
2016-07-28 22:34:15,967 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService.
2016-07-28 22:34:15,977 INFO  org.apache.flink.yarn.YarnTaskManager                         - Trying to register at JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2016-07-28 22:34:16,187 INFO  org.apache.flink.yarn.YarnTaskManager                         - Successful registration at JobManager (akka.tcp://flink@38.111.30.148:59857/user/jobmanager), starting network stack and library cache.
2016-07-28 22:34:16,445 INFO  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful initialization (took 77 ms).
2016-07-28 22:34:16,520 INFO  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful initialization (took 74 ms). Listening on SocketAddress /38.111.30.146:40641.
2016-07-28 22:34:16,521 INFO  org.apache.flink.yarn.YarnTaskManager                         - Determined BLOB server address to be /38.111.30.148:46326. Starting BLOB cache.
2016-07-28 22:34:16,551 INFO  org.apache.flink.runtime.blob.BlobCache                       - Created BLOB cache storage directory /tmp/blobStore-6505c99f-30f2-4a6f-a986-6bffccc0ddf9
2016-07-28 22:34:23,537 INFO  org.apache.flink.yarn.YarnTaskManager                         - Received task Source: Custom Source -> Sink: Unnamed (1/1)
2016-07-28 22:34:23,545 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Custom Source -> Sink: Unnamed (1/1)
2016-07-28 22:34:23,546 INFO  org.apache.flink.runtime.blob.BlobCache                       - Downloading d44873ce558492ab4d65fb4f9ca3d2da3090041f from /38.111.30.148:46326
2016-07-28 22:34:24,528 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Custom Source -> Sink: Unnamed (1/1) [DEPLOYING]
2016-07-28 22:34:24,531 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Sink: Unnamed (1/1) switched to RUNNING
2016-07-28 22:34:24,664 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - Using user-defined state backend: MemoryStateBackend (data in heap memory / checkpoints to JobManager)
2016-07-28 22:34:24,675 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - Using user-defined state backend: MemoryStateBackend (data in heap memory / checkpoints to JobManager)
2016-07-28 22:34:24,743 INFO  org.apache.kafka.clients.consumer.ConsumerConfig              - ConsumerConfig values: 
    metric.reporters = []
    metadata.max.age.ms = 300000
    value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
    group.id = 
    partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
    reconnect.backoff.ms = 50
    sasl.kerberos.ticket.renew.window.factor = 0.8
    max.partition.fetch.bytes = 1048576
    bootstrap.servers = [node01:6667]
    retry.backoff.ms = 100
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    ssl.keystore.type = JKS
    ssl.trustmanager.algorithm = PKIX
    enable.auto.commit = true
    ssl.key.password = null
    fetch.max.wait.ms = 500
    sasl.kerberos.min.time.before.relogin = 60000
    connections.max.idle.ms = 540000
    ssl.truststore.password = null
    session.timeout.ms = 30000
    metrics.num.samples = 2
    client.id = 
    ssl.endpoint.identification.algorithm = null
    key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
    ssl.protocol = TLS
    check.crcs = true
    request.timeout.ms = 40000
    ssl.provider = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.keystore.location = null
    heartbeat.interval.ms = 3000
    auto.commit.interval.ms = 5000
    receive.buffer.bytes = 32768
    ssl.cipher.suites = null
    ssl.truststore.type = JKS
    security.protocol = PLAINTEXT
    ssl.truststore.location = null
    ssl.keystore.password = null
    ssl.keymanager.algorithm = SunX509
    metrics.sample.window.ms = 30000
    fetch.min.bytes = 1
    send.buffer.bytes = 131072
    auto.offset.reset = earliest

2016-07-28 22:34:24,886 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.9.0.1
2016-07-28 22:34:24,886 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 23c69d62a0cabf06
2016-07-28 22:34:39,491 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2016-07-28 22:34:39,524 INFO  org.apache.flink.runtime.blob.BlobCache                       - Shutting down BlobCache
2016-07-28 22:34:39,655 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-96e84170-f59d-4316-bda1-d2c2810fdcdf
2016-07-28 22:34:39,656 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-996962ff-d621-4285-a4d2-7eb9e846ee11
2016-07-28 22:34:39,656 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-4b2cc5e6-110c-43aa-a1b1-620058e12a94
2016-07-28 22:34:39,656 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-a9c9d249-da5c-4c3c-a208-06603df1e966
End of LogType:taskmanager.log

LogType:taskmanager.out
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:0
Log Contents:
End of LogType:taskmanager.out

Container: container_e05_1467433388200_0136_01_000001 on node05.mycluster.com_45454
============================================================================================
LogType:jobmanager.err
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:0
Log Contents:
End of LogType:jobmanager.err

LogType:jobmanager.log
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:53981
Log Contents:
2016-07-28 22:33:59,613 INFO  org.apache.flink.yarn.ApplicationMaster                       - YARN daemon runs as yarn setting user to execute Flink ApplicationMaster/JobManager to user
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       - --------------------------------------------------------------------------------
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Starting YARN ApplicationMaster/JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC)
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Current user: yarn
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.60-b23
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Maximum heap size: 377 MiBytes
2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster                       -  JAVA_HOME: /usr/jdk64/jdk1.8.0_60
2016-07-28 22:33:59,632 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Hadoop version: 2.7.2
2016-07-28 22:33:59,632 INFO  org.apache.flink.yarn.ApplicationMaster                       -  JVM Options:
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -     -Xmx424M
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -     -Dlog.file=/mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.log
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -     -Dlogback.configurationFile=file:logback.xml
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -     -Dlog4j.configuration=file:log4j.properties
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Program Arguments: (none)
2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster                       -  Classpath: /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar
2016-07-28 22:33:59,634 INFO  org.apache.flink.yarn.ApplicationMaster                       - --------------------------------------------------------------------------------
2016-07-28 22:33:59,635 INFO  org.apache.flink.yarn.ApplicationMaster                       - Registered UNIX signal handlers for [TERM, HUP, INT]
2016-07-28 22:33:59,649 INFO  org.apache.flink.yarn.ApplicationMaster                       - Loading config from: /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001.
2016-07-28 22:33:59,720 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager
2016-07-28 22:33:59,722 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor system at 38.111.30.148:59857
2016-07-28 22:34:00,262 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2016-07-28 22:34:00,326 INFO  Remoting                                                      - Starting remoting
2016-07-28 22:34:00,534 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@38.111.30.148:59857]
2016-07-28 22:34:00,541 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManger web frontend
2016-07-28 22:34:00,545 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink' as root namespace.
2016-07-28 22:34:00,674 INFO  org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:host.name=node05.mycluster.com
2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.version=1.8.0_60
2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.vendor=Oracle Corporation
2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.class.path=/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.library.path=::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.io.tmpdir=/tmp
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:java.compiler=<NA>
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.name=Linux
2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.arch=amd64
2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:os.version=3.13.0-32-generic
2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.name=yarn
2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.home=/home/yarn
2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper                                - Client environment:user.dir=/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001
2016-07-28 22:34:00,687 INFO  org.apache.zookeeper.ZooKeeper                                - Initiating client connection, connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a1dd365
2016-07-28 22:34:00,714 INFO  org.apache.zookeeper.ClientCnxn                               - Opening socket connection to server node01.mycluster.com/38.111.30.144:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 22:34:00,720 INFO  org.apache.zookeeper.ClientCnxn                               - Socket connection established to node01.mycluster.com/38.111.30.144:2181, initiating session
2016-07-28 22:34:00,739 INFO  org.apache.zookeeper.ClientCnxn                               - Session establishment complete on server node01.mycluster.com/38.111.30.144:2181, sessionid = 0x15624f916ae08d2, negotiated timeout = 40000
2016-07-28 22:34:00,744 INFO  org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2016-07-28 22:34:01,773 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of JobManager log file: /mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.log
2016-07-28 22:34:01,773 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of JobManager stdout file: /mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.out
2016-07-28 22:34:01,802 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory /tmp/flink-web-fbe7069f-8e03-4535-b8d9-71db4f51b298 for the web interface files
2016-07-28 22:34:01,812 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory /tmp/flink-web-upload-1fecf32c-3b86-4d95-ae7b-25def122c3d4 for web frontend JAR file uploads
2016-07-28 22:34:02,113 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend listening at 0:0:0:0:0:0:0:0:48590
2016-07-28 22:34:02,114 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor
2016-07-28 22:34:02,125 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-de21570f-ca07-45a0-ae82-af4041390d86
2016-07-28 22:34:03,079 INFO  org.apache.flink.runtime.blob.FileSystemBlobStore             - Created blob directory hdfs:///flink/recovery/blob.
2016-07-28 22:34:03,080 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:46326 - max concurrent requests: 50 - max backlog: 1000
2016-07-28 22:34:03,092 INFO  org.apache.flink.runtime.util.ZooKeeperUtils                  - Using '/flink' as root namespace.
2016-07-28 22:34:03,094 INFO  org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl  - Starting
2016-07-28 22:34:03,095 INFO  org.apache.zookeeper.ZooKeeper                                - Initiating client connection, connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@14fc5d40
2016-07-28 22:34:03,097 INFO  org.apache.zookeeper.ClientCnxn                               - Opening socket connection to server node03.mycluster.com/38.111.30.146:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-28 22:34:03,099 INFO  org.apache.zookeeper.ClientCnxn                               - Socket connection established to node03.mycluster.com/38.111.30.146:2181, initiating session
2016-07-28 22:34:03,119 INFO  org.apache.zookeeper.ClientCnxn                               - Session establishment complete on server node03.mycluster.com/38.111.30.146:2181, sessionid = 0x3556f328b2602ff, negotiated timeout = 40000
2016-07-28 22:34:03,119 INFO  org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager  - State change: CONNECTED
2016-07-28 22:34:03,147 INFO  org.apache.flink.runtime.checkpoint.SavepointStoreFactory     - Using filesystem savepoint backend (root path: hdfs:///flink/savepoints).
2016-07-28 22:34:03,158 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started memory archivist akka://flink/user/archive
2016-07-28 22:34:03,161 INFO  org.apache.flink.yarn.YarnJobManager                          - Starting JobManager at akka.tcp://<a href="http://flink@38.111.30.148:59857/user/jobmanager.
2016-07-28">flink@38.111.30.148:59857/user/jobmanager.
2016-07-28 22:34:03,161 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService.
2016-07-28 22:34:03,166 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting with JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager on port 48590
2016-07-28 22:34:03,167 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService.
2016-07-28 22:34:03,176 INFO  org.apache.flink.yarn.ApplicationMaster                       - Generate configuration file for application master.
2016-07-28 22:34:03,206 INFO  org.apache.flink.yarn.ApplicationMaster                       - Starting YARN session on Job Manager.
2016-07-28 22:34:03,206 INFO  org.apache.flink.yarn.ApplicationMaster                       - Application Master properly initiated. Awaiting termination of actor system.
2016-07-28 22:34:03,214 INFO  org.apache.flink.yarn.YarnJobManager                          - Start yarn session.
2016-07-28 22:34:03,273 INFO  org.apache.flink.yarn.YarnJobManager                          - Yarn session with 1 TaskManagers. Tolerating 1 failed TaskManagers
2016-07-28 22:34:03,302 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at <a href="http://node02.mycluster.com/38.111.30.145:8030
2016-07-28">node02.mycluster.com/38.111.30.145:8030
2016-07-28 22:34:03,324 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - yarn.client.max-cached-nodemanagers-proxies : 0
2016-07-28 22:34:03,325 INFO  org.apache.flink.yarn.YarnJobManager                          - Registering ApplicationMaster with tracking url http://node05.mycluster.com:48590.
2016-07-28 22:34:03,771 INFO  org.apache.flink.yarn.YarnJobManager                          - Retrieved 0 TaskManagers from previous attempts.
2016-07-28 22:34:03,776 INFO  org.apache.flink.yarn.YarnJobManager                          - Requesting initial TaskManager container 0.
2016-07-28 22:34:03,811 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink-conf-modified.yaml to hdfs://<a href="http://node01.mycluster.com:8020/user/user/.flink/application_1467433388200_0136/flink-conf-modified.yaml
2016-07-28">node01.mycluster.com:8020/user/user/.flink/application_1467433388200_0136/flink-conf-modified.yaml
2016-07-28 22:34:04,443 INFO  org.apache.flink.yarn.YarnJobManager                          - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "node01.mycluster.com" port: 8020 file: "/user/user/.flink/application_1467433388200_0136/flink-conf-modified.yaml" } size: 5593 timestamp: 1469745244302 type: FILE visibility: APPLICATION
2016-07-28 22:34:04,449 INFO  org.apache.flink.yarn.YarnJobManager                          - Create container launch context.
2016-07-28 22:34:04,459 INFO  org.apache.flink.yarn.YarnJobManager                          - Starting TM with command=$JAVA_HOME/bin/java -Xms1448m -Xmx1448m -XX:MaxDirectMemorySize=1448m  -Dlog.file="<LOG_DIR>/taskmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err
2016-07-28 22:34:04,463 INFO  org.apache.flink.yarn.YarnJobManager                          - JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager was granted leadership with leader session ID Some(3d2a8629-081e-44bc-8c38-446b510d5909).
2016-07-28 22:34:04,504 INFO  org.apache.flink.yarn.YarnJobManager                          - Delaying recovery of all jobs by 10000 milliseconds.
2016-07-28 22:34:04,511 INFO  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader reachable under akka.tcp://<a href="http://flink@38.111.30.148:59857/user/jobmanager:3d2a8629-081e-44bc-8c38-446b510d5909.
2016-07-28">flink@38.111.30.148:59857/user/jobmanager:3d2a8629-081e-44bc-8c38-446b510d5909.
2016-07-28 22:34:04,878 INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Received new token for : node03.mycluster.com:45454
2016-07-28 22:34:04,900 INFO  org.apache.flink.yarn.YarnJobManager                          - Got new container for allocation: container_e05_1467433388200_0136_01_000002
2016-07-28 22:34:04,900 INFO  org.apache.flink.yarn.YarnJobManager                          - The user requested 1 containers, 0 running. 1 containers missing
2016-07-28 22:34:04,901 INFO  org.apache.flink.yarn.YarnJobManager                          - 1 containers already allocated by YARN. Starting...
2016-07-28 22:34:04,902 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - Opening proxy : node03.mycluster.com:45454
2016-07-28 22:34:05,149 INFO  org.apache.flink.yarn.YarnJobManager                          - Launching container (container_e05_1467433388200_0136_01_000002 on host node03.mycluster.com).
2016-07-28 22:34:07,375 INFO  org.apache.flink.yarn.YarnJobManager                          - Register akka.tcp://flink@38.111.30.144:45229/user/applicationClient as client.
2016-07-28 22:34:14,524 INFO  org.apache.flink.yarn.YarnJobManager                          - Attempting to recover all jobs.
2016-07-28 22:34:14,527 INFO  org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - No job graph to recover.
2016-07-28 22:34:14,528 INFO  org.apache.flink.yarn.YarnJobManager                          - Re-submitting 0 job graphs.
2016-07-28 22:34:16,168 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at node03 (akka.tcp://flink@38.111.30.146:48700/user/taskmanager) as 1d52e723ce440d6181b6e0e98627bfd2. Current number of registered hosts is 1. Current number of alive task slots is 1.
2016-07-28 22:34:23,098 INFO  org.apache.flink.yarn.YarnJobManager                          - Submitting job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job).
2016-07-28 22:34:23,100 INFO  org.apache.flink.yarn.YarnJobManager                          - Using restart strategy NoRestartStrategy for 9cdd920b36bec849689a72026402a333.
2016-07-28 22:34:23,387 INFO  org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - Added SubmittedJobGraph(JobGraph(jobId: 9cdd920b36bec849689a72026402a333), JobInfo(client: Actor[akka.tcp://flink@127.0.0.1:38212/user/$a#-2098822004] (EXECUTION_RESULT_AND_STATE_CHANGES), start: 1469745263098)) to ZooKeeper.
2016-07-28 22:34:23,395 INFO  org.apache.flink.yarn.YarnJobManager                          - Scheduling job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job).
2016-07-28 22:34:23,397 INFO  org.apache.flink.yarn.YarnJobManager                          - Status of job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job) changed to RUNNING.
2016-07-28 22:34:23,400 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282) switched from CREATED to SCHEDULED
2016-07-28 22:34:23,423 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282) switched from SCHEDULED to DEPLOYING
2016-07-28 22:34:23,424 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Custom Source -> Sink: Unnamed (1/1) (attempt #0) to node03
2016-07-28 22:34:24,554 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282) switched from DEPLOYING to RUNNING
2016-07-28 22:34:40,021 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@38.111.30.146:48700] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager                          - Container container_e05_1467433388200_0136_01_000002 is completed with diagnostics: Container [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779 /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m -XX:MaxDirectMemorySize=1448m -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 
    |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m -XX:MaxDirectMemorySize=1448m  -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out 2> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

2016-07-28 22:34:40,400 INFO  org.apache.flink.yarn.YarnJobManager                          - Container container_e05_1467433388200_0136_01_000002 was a running container. Total failed containers 1.
2016-07-28 22:34:40,404 ERROR org.apache.flink.yarn.YarnJobManager                          - Stopping YARN session because the number of failed containers (1) exceeded the maximum failed container count (1). This number is controlled by the 'yarn.maximum-failed-containers' configuration setting. By default its the number of requested containers
2016-07-28 22:34:40,405 INFO  org.apache.flink.yarn.YarnJobManager                          - The user requested 1 containers, 0 running. 1 containers missing
2016-07-28 22:34:40,406 INFO  org.apache.flink.yarn.YarnJobManager                          - There are 1 containers missing. 0 are already requested. Requesting 1 additional container(s) from YARN. Reallocation of failed containers is enabled=true ('yarn.reallocate-failed')
2016-07-28 22:34:40,408 INFO  org.apache.flink.yarn.YarnJobManager                          - Requested additional container from YARN. Pending requests 1.
2016-07-28 22:34:40,410 INFO  org.apache.flink.yarn.YarnJobManager                          - Stopping YARN JobManager with status FAILED and diagnostic Stopping YARN session because the number of failed containers (1) exceeded the maximum failed container count (1). This number is controlled by the 'yarn.maximum-failed-containers' configuration setting. By default its the number of requested containers.
2016-07-28 22:34:40,419 INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Waiting for application to be successfully unregistered.
2016-07-28 22:34:40,525 INFO  org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl  - Interrupted while waiting for queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287)
2016-07-28 22:34:40,528 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - Opening proxy : node03.mycluster.com:45454
2016-07-28 22:34:40,647 INFO  org.apache.flink.yarn.YarnJobManager                          - Stopping JobManager akka.tcp://<a href="http://flink@38.111.30.148:59857/user/jobmanager.
2016-07-28">flink@38.111.30.148:59857/user/jobmanager.
2016-07-28 22:34:40,883 INFO  org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - Removed job graph 9cdd920b36bec849689a72026402a333 from ZooKeeper.
2016-07-28 22:34:40,911 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282) switched from RUNNING to CANCELING
2016-07-28 22:34:40,923 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Stopping ZooKeeperLeaderElectionService.
2016-07-28 22:34:40,955 INFO  org.apache.zookeeper.ClientCnxn                               - EventThread shut down
2016-07-28 22:34:40,955 INFO  org.apache.zookeeper.ZooKeeper                                - Session: 0x3556f328b2602ff closed
2016-07-28 22:34:40,961 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282) switched from CANCELING to CANCELED
2016-07-28 22:34:41,145 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:46326
2016-07-28 22:34:41,152 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
2016-07-28 22:34:41,153 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
2016-07-28 22:34:41,185 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
2016-07-28 22:34:41,199 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping ZooKeeperLeaderRetrievalService.
2016-07-28 22:34:41,202 INFO  org.apache.flink.yarn.YarnJobManager                          - Shutdown completed. Stopping JVM.
2016-07-28 22:34:41,208 INFO  org.apache.zookeeper.ZooKeeper                                - Session: 0x15624f916ae08d2 closed
2016-07-28 22:34:41,208 INFO  org.apache.zookeeper.ClientCnxn                               - EventThread shut down
2016-07-28 22:34:41,209 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web dashboard root cache directory /tmp/flink-web-fbe7069f-8e03-4535-b8d9-71db4f51b298
2016-07-28 22:34:41,209 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web dashboard jar upload directory /tmp/flink-web-upload-1fecf32c-3b86-4d95-ae7b-25def122c3d4
2016-07-28 22:34:41,217 INFO  org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator  - Shutting down stack trace sample coordinator.
End of LogType:jobmanager.log

LogType:jobmanager.out
Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
LogLength:0
Log Contents:
End of LogType:jobmanager.out
​



On Fri, Jul 29, 2016 at 11:19 AM, Jack Huang <[hidden email]> wrote:
Hi Max,

Each events are only a few hundred bytes. I am reading from a Kafka topic from some offset in the past, so the events should be flowing in as fast as Flink can process them.

The entire YARN task log, which contains both JobManager and TaskManager outputs, is attached.

Thanks a lot,
Jack


On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]> wrote:
Hi Jack,

Considering the type of job you're running, you shouldn't run out of
memory. Could it be that the events are quite large strings? It could
be that the TextOutputFormat doesn't write to disk fast enough and
accumulates memory. Actually, it doesn't perform regular flushing
which could be an issue.

I'm just guessing, we need to investigate further. Could you please
supply the entire JobManager log file output?

Thanks,
Max

On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
> Hi all,
>
> I am running a test Flink streaming task under YARN. It reads messages from
> a Kafka topic and writes them to local file system.
>
> object PricerEvent {
>     def main(args:Array[String]) {
>         val kafkaProp = new Properties()
>         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>         kafkaProp.setProperty("auto.offset.reset", "earliest")
>
>         val env = StreamExecutionEnvironment.getExecutionEnvironment
>         env.setStateBackend(new MemoryStateBackend)
>
>         val wins = env.addSource(new FlinkKafkaConsumer09[String]("events",
> new SimpleStringSchema, kafkaProp))
>         wins.writeAsText("/home/user/flink_out/" + new
> SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>
>         env.execute
>     }
> }
>
> With the command
>
> flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
> /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>
>
> The task runs fine for a moment and then terminates. I looked into the error
> log and found following out-of-memory error message:
>
> 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
> - Container container_e05_1467433388200_0136_01_000002 is completed with
> diagnostics: Container
> [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running
> beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory
> used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
> 2>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> I don't understand how it should fail due to out-of-memory error. I would
> expect the task to slow down if there are too many messages to process, but
> not failing altogether. I am not storing any states either.
>
> Does anyone know the reason and the way to fix/avoid this issue?
>
>
> Thanks,
> Jack
>
>
>
>
Maximilian Michels
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Maximilian Michels
Your job creates a lot of String objects which need to be garbage
collected. It could be that the JVM is not fast enough and Yarn kills
the JVM for consuming too much memory.

You can try two things:

1) Give the task manager more memory
2) Increase the Yarn heap cutoff ratio (e.g yarn.heap-cutoff-ratio: 0.4)

If the error still occurs then we need to investigate further.

Thanks,
Max

On Mon, Aug 1, 2016 at 7:48 PM, Jack Huang <[hidden email]> wrote:

> Hi Max, here is the YARN task log. Searching for "limit" will bring you to
> the error line.
>
> Container: container_e05_1467433388200_0136_01_000002 on
> node03.mycluster.com_45454
> ============================================================================================
> LogType:taskmanager.err
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:0
> Log Contents:
> End of LogType:taskmanager.err
>
> LogType:taskmanager.log
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:49515
> Log Contents:
> 2016-07-28 22:34:12,861 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -
> --------------------------------------------------------------------------------
> 2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Starting YARN TaskManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @
> 13:40:33 UTC)
> 2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Current user: yarn
> 2016-07-28 22:34:12,871 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation -
> 1.8/25.60-b23
> 2016-07-28 22:34:12,872 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Maximum heap size: 1388 MiBytes
> 2016-07-28 22:34:12,872 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  JAVA_HOME: /usr/jdk64/jdk1.8.0_60
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Hadoop version: 2.7.2
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  JVM Options:
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     -Xms1448m
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     -Xmx1448m
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     -XX:MaxDirectMemorySize=1448m
> 2016-07-28 22:34:12,874 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     -Dlogback.configurationFile=file:logback.xml
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     -Dlog4j.configuration=file:log4j.properties
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Program Arguments:
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     --configDir
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -     .
> 2016-07-28 22:34:12,875 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -  Classpath:
> /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar
> 2016-07-28 22:34:12,877 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> -
> --------------------------------------------------------------------------------
> 2016-07-28 22:34:12,878 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> - Registered UNIX signal handlers for [TERM, HUP, INT]
> 2016-07-28 22:34:13,098 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Loading
> configuration from .
> 2016-07-28 22:34:13,111 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> - Setting directories for temporary file
> /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136,/mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136
> 2016-07-28 22:34:13,111 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> - YARN daemon runs as 'yarn' setting user to execute Flink TaskManager to
> 'user'
> 2016-07-28 22:34:13,126 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
> - Using '/flink' as root namespace.
> 2016-07-28 22:34:13,224 INFO
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
> - Starting
> 2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT
> 2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:host.name=node03.mycluster.com
> 2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.version=1.8.0_60
> 2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.vendor=Oracle Corporation
> 2016-07-28 22:34:13,242 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
> 2016-07-28 22:34:13,243 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:java.class.path=/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:java.library.path=::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.io.tmpdir=/tmp
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.compiler=<NA>
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.name=Linux
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.arch=amd64
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.version=3.13.0-32-generic
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:user.name=yarn
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:user.home=/home/yarn
> 2016-07-28 22:34:13,244 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:user.dir=/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002
> 2016-07-28 22:34:13,245 INFO  org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@5f8e8a9d
> 2016-07-28 22:34:13,314 INFO  org.apache.zookeeper.ClientCnxn
> - Opening socket connection to server
> node03.mycluster.com/38.111.30.146:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2016-07-28 22:34:13,327 INFO  org.apache.zookeeper.ClientCnxn
> - Socket connection established to node03.mycluster.com/38.111.30.146:2181,
> initiating session
> 2016-07-28 22:34:13,354 INFO  org.apache.zookeeper.ClientCnxn
> - Session establishment complete on server
> node03.mycluster.com/38.111.30.146:2181, sessionid = 0x3556f328b260300,
> negotiated timeout = 40000
> 2016-07-28 22:34:13,360 INFO
> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager
> - State change: CONNECTED
> 2016-07-28 22:34:14,430 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  -
> Starting ZooKeeperLeaderRetrievalService.
> 2016-07-28 22:34:14,436 INFO
> org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to
> select the network interface and address to use by connecting to the leading
> JobManager.
> 2016-07-28 22:34:14,437 INFO
> org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager
> will try to connect for 10000 milliseconds before falling back to heuristics
> 2016-07-28 22:34:14,549 INFO  org.apache.flink.runtime.net.ConnectionUtils
> - Retrieved new target address /38.111.30.148:59857.
> 2016-07-28 22:34:14,578 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  -
> Stopping ZooKeeperLeaderRetrievalService.
> 2016-07-28 22:34:14,590 INFO  org.apache.zookeeper.ZooKeeper
> - Session: 0x3556f328b260300 closed
> 2016-07-28 22:34:14,590 INFO  org.apache.zookeeper.ClientCnxn
> - EventThread shut down
> 2016-07-28 22:34:14,597 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager
> will use hostname/address 'node03' (38.111.30.146) for communication.
> 2016-07-28 22:34:14,598 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting
> TaskManager
> 2016-07-28 22:34:14,599 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting
> TaskManager actor system at 38.111.30.146:0
> 2016-07-28 22:34:15,001 INFO  akka.event.slf4j.Slf4jLogger
> - Slf4jLogger started
> 2016-07-28 22:34:15,046 INFO  Remoting
> - Starting remoting
> 2016-07-28 22:34:15,353 INFO  Remoting
> - Remoting started; listening on addresses
> :[akka.tcp://flink@38.111.30.146:48700]
> 2016-07-28 22:34:15,360 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Starting
> TaskManager actor
> 2016-07-28 22:34:15,369 INFO
> org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig
> [server address: node03/38.111.30.146, server port: 40641, memory segment
> size (bytes): 32768, transport type: NIO, number of server threads: 1
> (manual), number of client threads: 1 (manual), server connect backlog: 0
> (use Netty's default), client connect timeout (sec): 120, send/receive
> buffer size (bytes): 0 (use Netty's default)]
> 2016-07-28 22:34:15,372 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Messages
> between TaskManager and JobManager have a max timeout of 10000 milliseconds
> 2016-07-28 22:34:15,378 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
> file directory
> '/mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136':
> total 2734 GB, usable 2055 GB (75.16% usable)
> 2016-07-28 22:34:15,379 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
> file directory
> '/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136':
> total 2750 GB, usable 2064 GB (75.05% usable)
> 2016-07-28 22:34:15,379 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
> file directory
> '/mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136':
> total 2750 GB, usable 2233 GB (81.20% usable)
> 2016-07-28 22:34:15,380 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
> file directory
> '/mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136':
> total 2750 GB, usable 2466 GB (89.67% usable)
> 2016-07-28 22:34:15,416 INFO
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 64
> MB for network buffer pool (number of memory segments: 2048, bytes per
> segment: 32768).
> 2016-07-28 22:34:15,599 INFO
> org.apache.flink.runtime.taskmanager.TaskManager              - Limiting
> managed memory to 0.7 of the currently free heap space (922 MB), memory will
> be allocated lazily.
> 2016-07-28 22:34:15,626 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> uses directory
> /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-96e84170-f59d-4316-bda1-d2c2810fdcdf
> for spill files.
> 2016-07-28 22:34:15,626 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> uses directory
> /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-996962ff-d621-4285-a4d2-7eb9e846ee11
> for spill files.
> 2016-07-28 22:34:15,626 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> uses directory
> /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-4b2cc5e6-110c-43aa-a1b1-620058e12a94
> for spill files.
> 2016-07-28 22:34:15,627 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> uses directory
> /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-a9c9d249-da5c-4c3c-a208-06603df1e966
> for spill files.
> 2016-07-28 22:34:15,630 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
> - Using '/flink' as root namespace.
> 2016-07-28 22:34:15,630 INFO
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
> - Starting
> 2016-07-28 22:34:15,631 INFO  org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@4678a2eb
> 2016-07-28 22:34:15,634 INFO  org.apache.zookeeper.ClientCnxn
> - Opening socket connection to server
> node02.mycluster.com/38.111.30.145:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2016-07-28 22:34:15,634 INFO  org.apache.zookeeper.ClientCnxn
> - Socket connection established to node02.mycluster.com/38.111.30.145:2181,
> initiating session
> 2016-07-28 22:34:15,656 INFO  org.apache.zookeeper.ClientCnxn
> - Session establishment complete on server
> node02.mycluster.com/38.111.30.145:2181, sessionid = 0x255bc6698ff025f,
> negotiated timeout = 40000
> 2016-07-28 22:34:15,656 INFO
> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager
> - State change: CONNECTED
> 2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
> 2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
> 2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
> 2016-07-28 22:34:15,667 INFO  org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-dist-cache-cba06958-4240-4070-b611-d4687e1bbea0
> 2016-07-28 22:34:15,964 INFO  org.apache.flink.yarn.YarnTaskManager
> - Starting TaskManager actor at akka://flink/user/taskmanager#-1203652007.
> 2016-07-28 22:34:15,964 INFO  org.apache.flink.yarn.YarnTaskManager
> - TaskManager data connection information: node03.mycluster.com
> (dataPort=40641)
> 2016-07-28 22:34:15,965 INFO  org.apache.flink.yarn.YarnTaskManager
> - TaskManager has 1 task slot(s).
> 2016-07-28 22:34:15,967 INFO  org.apache.flink.yarn.YarnTaskManager
> - Memory usage stats: [HEAP: 123/1388/1388 MB, NON HEAP: 33/34/-1 MB
> (used/committed/max)]
> 2016-07-28 22:34:15,967 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  -
> Starting ZooKeeperLeaderRetrievalService.
> 2016-07-28 22:34:15,977 INFO  org.apache.flink.yarn.YarnTaskManager
> - Trying to register at JobManager
> akka.tcp://flink@38.111.30.148:59857/user/jobmanager (attempt 1, timeout:
> 500 milliseconds)
> 2016-07-28 22:34:16,187 INFO  org.apache.flink.yarn.YarnTaskManager
> - Successful registration at JobManager
> (akka.tcp://flink@38.111.30.148:59857/user/jobmanager), starting network
> stack and library cache.
> 2016-07-28 22:34:16,445 INFO
> org.apache.flink.runtime.io.network.netty.NettyClient         - Successful
> initialization (took 77 ms).
> 2016-07-28 22:34:16,520 INFO
> org.apache.flink.runtime.io.network.netty.NettyServer         - Successful
> initialization (took 74 ms). Listening on SocketAddress
> /38.111.30.146:40641.
> 2016-07-28 22:34:16,521 INFO  org.apache.flink.yarn.YarnTaskManager
> - Determined BLOB server address to be /38.111.30.148:46326. Starting BLOB
> cache.
> 2016-07-28 22:34:16,551 INFO  org.apache.flink.runtime.blob.BlobCache
> - Created BLOB cache storage directory
> /tmp/blobStore-6505c99f-30f2-4a6f-a986-6bffccc0ddf9
> 2016-07-28 22:34:23,537 INFO  org.apache.flink.yarn.YarnTaskManager
> - Received task Source: Custom Source -> Sink: Unnamed (1/1)
> 2016-07-28 22:34:23,545 INFO  org.apache.flink.runtime.taskmanager.Task
> - Loading JAR files for task Source: Custom Source -> Sink: Unnamed (1/1)
> 2016-07-28 22:34:23,546 INFO  org.apache.flink.runtime.blob.BlobCache
> - Downloading d44873ce558492ab4d65fb4f9ca3d2da3090041f from
> /38.111.30.148:46326
> 2016-07-28 22:34:24,528 INFO  org.apache.flink.runtime.taskmanager.Task
> - Registering task at network: Source: Custom Source -> Sink: Unnamed (1/1)
> [DEPLOYING]
> 2016-07-28 22:34:24,531 INFO  org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source -> Sink: Unnamed (1/1) switched to RUNNING
> 2016-07-28 22:34:24,664 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask           - Using
> user-defined state backend: MemoryStateBackend (data in heap memory /
> checkpoints to JobManager)
> 2016-07-28 22:34:24,675 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask           - Using
> user-defined state backend: MemoryStateBackend (data in heap memory /
> checkpoints to JobManager)
> 2016-07-28 22:34:24,743 INFO
> org.apache.kafka.clients.consumer.ConsumerConfig              -
> ConsumerConfig values:
>     metric.reporters = []
>     metadata.max.age.ms = 300000
>     value.deserializer = class
> org.apache.kafka.common.serialization.ByteArrayDeserializer
>     group.id =
>     partition.assignment.strategy =
> [org.apache.kafka.clients.consumer.RangeAssignor]
>     reconnect.backoff.ms = 50
>     sasl.kerberos.ticket.renew.window.factor = 0.8
>     max.partition.fetch.bytes = 1048576
>     bootstrap.servers = [node01:6667]
>     retry.backoff.ms = 100
>     sasl.kerberos.kinit.cmd = /usr/bin/kinit
>     sasl.kerberos.service.name = null
>     sasl.kerberos.ticket.renew.jitter = 0.05
>     ssl.keystore.type = JKS
>     ssl.trustmanager.algorithm = PKIX
>     enable.auto.commit = true
>     ssl.key.password = null
>     fetch.max.wait.ms = 500
>     sasl.kerberos.min.time.before.relogin = 60000
>     connections.max.idle.ms = 540000
>     ssl.truststore.password = null
>     session.timeout.ms = 30000
>     metrics.num.samples = 2
>     client.id =
>     ssl.endpoint.identification.algorithm = null
>     key.deserializer = class
> org.apache.kafka.common.serialization.ByteArrayDeserializer
>     ssl.protocol = TLS
>     check.crcs = true
>     request.timeout.ms = 40000
>     ssl.provider = null
>     ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>     ssl.keystore.location = null
>     heartbeat.interval.ms = 3000
>     auto.commit.interval.ms = 5000
>     receive.buffer.bytes = 32768
>     ssl.cipher.suites = null
>     ssl.truststore.type = JKS
>     security.protocol = PLAINTEXT
>     ssl.truststore.location = null
>     ssl.keystore.password = null
>     ssl.keymanager.algorithm = SunX509
>     metrics.sample.window.ms = 30000
>     fetch.min.bytes = 1
>     send.buffer.bytes = 131072
>     auto.offset.reset = earliest
>
> 2016-07-28 22:34:24,886 INFO  org.apache.kafka.common.utils.AppInfoParser
> - Kafka version : 0.9.0.1
> 2016-07-28 22:34:24,886 INFO  org.apache.kafka.common.utils.AppInfoParser
> - Kafka commitId : 23c69d62a0cabf06
> 2016-07-28 22:34:39,491 INFO  org.apache.flink.yarn.YarnTaskManagerRunner
> - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2016-07-28 22:34:39,524 INFO  org.apache.flink.runtime.blob.BlobCache
> - Shutting down BlobCache
> 2016-07-28 22:34:39,655 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> removed spill file directory
> /mnt/a/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-96e84170-f59d-4316-bda1-d2c2810fdcdf
> 2016-07-28 22:34:39,656 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> removed spill file directory
> /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-996962ff-d621-4285-a4d2-7eb9e846ee11
> 2016-07-28 22:34:39,656 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> removed spill file directory
> /mnt/c/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-4b2cc5e6-110c-43aa-a1b1-620058e12a94
> 2016-07-28 22:34:39,656 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager
> removed spill file directory
> /mnt/d/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/flink-io-a9c9d249-da5c-4c3c-a208-06603df1e966
> End of LogType:taskmanager.log
>
> LogType:taskmanager.out
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:0
> Log Contents:
> End of LogType:taskmanager.out
>
> Container: container_e05_1467433388200_0136_01_000001 on
> node05.mycluster.com_45454
> ============================================================================================
> LogType:jobmanager.err
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:0
> Log Contents:
> End of LogType:jobmanager.err
>
> LogType:jobmanager.log
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:53981
> Log Contents:
> 2016-07-28 22:33:59,613 INFO  org.apache.flink.yarn.ApplicationMaster
> - YARN daemon runs as yarn setting user to execute Flink
> ApplicationMaster/JobManager to user
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -
> --------------------------------------------------------------------------------
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Starting YARN ApplicationMaster/JobManager (Version: 1.0.1, Rev:4afa401,
> Date:31.03.2016 @ 13:40:33 UTC)
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Current user: yarn
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation -
> 1.8/25.60-b23
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Maximum heap size: 377 MiBytes
> 2016-07-28 22:33:59,631 INFO  org.apache.flink.yarn.ApplicationMaster
> -  JAVA_HOME: /usr/jdk64/jdk1.8.0_60
> 2016-07-28 22:33:59,632 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Hadoop version: 2.7.2
> 2016-07-28 22:33:59,632 INFO  org.apache.flink.yarn.ApplicationMaster
> -  JVM Options:
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -     -Xmx424M
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -
> -Dlog.file=/mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.log
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -     -Dlogback.configurationFile=file:logback.xml
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -     -Dlog4j.configuration=file:log4j.properties
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Program Arguments: (none)
> 2016-07-28 22:33:59,633 INFO  org.apache.flink.yarn.ApplicationMaster
> -  Classpath:
> /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar
> 2016-07-28 22:33:59,634 INFO  org.apache.flink.yarn.ApplicationMaster
> -
> --------------------------------------------------------------------------------
> 2016-07-28 22:33:59,635 INFO  org.apache.flink.yarn.ApplicationMaster
> - Registered UNIX signal handlers for [TERM, HUP, INT]
> 2016-07-28 22:33:59,649 INFO  org.apache.flink.yarn.ApplicationMaster
> - Loading config from:
> /mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001.
> 2016-07-28 22:33:59,720 INFO  org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManager
> 2016-07-28 22:33:59,722 INFO  org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManager actor system at 38.111.30.148:59857
> 2016-07-28 22:34:00,262 INFO  akka.event.slf4j.Slf4jLogger
> - Slf4jLogger started
> 2016-07-28 22:34:00,326 INFO  Remoting
> - Starting remoting
> 2016-07-28 22:34:00,534 INFO  Remoting
> - Remoting started; listening on addresses
> :[akka.tcp://flink@38.111.30.148:59857]
> 2016-07-28 22:34:00,541 INFO  org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManger web frontend
> 2016-07-28 22:34:00,545 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
> - Using '/flink' as root namespace.
> 2016-07-28 22:34:00,674 INFO
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
> - Starting
> 2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT
> 2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:host.name=node05.mycluster.com
> 2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.version=1.8.0_60
> 2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.vendor=Oracle Corporation
> 2016-07-28 22:34:00,684 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:java.class.path=/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink.jar:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-plugins-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.1.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.3.4.0-3485.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:java.library.path=::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native::/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.io.tmpdir=/tmp
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:java.compiler=<NA>
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.name=Linux
> 2016-07-28 22:34:00,685 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.arch=amd64
> 2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:os.version=3.13.0-32-generic
> 2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:user.name=yarn
> 2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper
> - Client environment:user.home=/home/yarn
> 2016-07-28 22:34:00,686 INFO  org.apache.zookeeper.ZooKeeper
> - Client
> environment:user.dir=/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001
> 2016-07-28 22:34:00,687 INFO  org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a1dd365
> 2016-07-28 22:34:00,714 INFO  org.apache.zookeeper.ClientCnxn
> - Opening socket connection to server
> node01.mycluster.com/38.111.30.144:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2016-07-28 22:34:00,720 INFO  org.apache.zookeeper.ClientCnxn
> - Socket connection established to node01.mycluster.com/38.111.30.144:2181,
> initiating session
> 2016-07-28 22:34:00,739 INFO  org.apache.zookeeper.ClientCnxn
> - Session establishment complete on server
> node01.mycluster.com/38.111.30.144:2181, sessionid = 0x15624f916ae08d2,
> negotiated timeout = 40000
> 2016-07-28 22:34:00,744 INFO
> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager
> - State change: CONNECTED
> 2016-07-28 22:34:01,773 INFO
> org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined
> location of JobManager log file:
> /mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.log
> 2016-07-28 22:34:01,773 INFO
> org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined
> location of JobManager stdout file:
> /mnt/d/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/jobmanager.out
> 2016-07-28 22:34:01,802 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
> directory /tmp/flink-web-fbe7069f-8e03-4535-b8d9-71db4f51b298 for the web
> interface files
> 2016-07-28 22:34:01,812 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
> directory /tmp/flink-web-upload-1fecf32c-3b86-4d95-ae7b-25def122c3d4 for web
> frontend JAR file uploads
> 2016-07-28 22:34:02,113 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend
> listening at 0:0:0:0:0:0:0:0:48590
> 2016-07-28 22:34:02,114 INFO  org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManager actor
> 2016-07-28 22:34:02,125 INFO  org.apache.flink.runtime.blob.BlobServer
> - Created BLOB server storage directory
> /tmp/blobStore-de21570f-ca07-45a0-ae82-af4041390d86
> 2016-07-28 22:34:03,079 INFO
> org.apache.flink.runtime.blob.FileSystemBlobStore             - Created blob
> directory hdfs:///flink/recovery/blob.
> 2016-07-28 22:34:03,080 INFO  org.apache.flink.runtime.blob.BlobServer
> - Started BLOB server at 0.0.0.0:46326 - max concurrent requests: 50 - max
> backlog: 1000
> 2016-07-28 22:34:03,092 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
> - Using '/flink' as root namespace.
> 2016-07-28 22:34:03,094 INFO
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
> - Starting
> 2016-07-28 22:34:03,095 INFO  org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@14fc5d40
> 2016-07-28 22:34:03,097 INFO  org.apache.zookeeper.ClientCnxn
> - Opening socket connection to server
> node03.mycluster.com/38.111.30.146:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2016-07-28 22:34:03,099 INFO  org.apache.zookeeper.ClientCnxn
> - Socket connection established to node03.mycluster.com/38.111.30.146:2181,
> initiating session
> 2016-07-28 22:34:03,119 INFO  org.apache.zookeeper.ClientCnxn
> - Session establishment complete on server
> node03.mycluster.com/38.111.30.146:2181, sessionid = 0x3556f328b2602ff,
> negotiated timeout = 40000
> 2016-07-28 22:34:03,119 INFO
> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager
> - State change: CONNECTED
> 2016-07-28 22:34:03,147 INFO
> org.apache.flink.runtime.checkpoint.SavepointStoreFactory     - Using
> filesystem savepoint backend (root path: hdfs:///flink/savepoints).
> 2016-07-28 22:34:03,158 INFO
> org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started
> memory archivist akka://flink/user/archive
> 2016-07-28 22:34:03,161 INFO  org.apache.flink.yarn.YarnJobManager
> - Starting JobManager at
> akka.tcp://flink@38.111.30.148:59857/user/jobmanager.
> 2016-07-28 22:34:03,161 INFO
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
> Starting ZooKeeperLeaderElectionService.
> 2016-07-28 22:34:03,166 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting
> with JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager on port
> 48590
> 2016-07-28 22:34:03,167 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  -
> Starting ZooKeeperLeaderRetrievalService.
> 2016-07-28 22:34:03,176 INFO  org.apache.flink.yarn.ApplicationMaster
> - Generate configuration file for application master.
> 2016-07-28 22:34:03,206 INFO  org.apache.flink.yarn.ApplicationMaster
> - Starting YARN session on Job Manager.
> 2016-07-28 22:34:03,206 INFO  org.apache.flink.yarn.ApplicationMaster
> - Application Master properly initiated. Awaiting termination of actor
> system.
> 2016-07-28 22:34:03,214 INFO  org.apache.flink.yarn.YarnJobManager
> - Start yarn session.
> 2016-07-28 22:34:03,273 INFO  org.apache.flink.yarn.YarnJobManager
> - Yarn session with 1 TaskManagers. Tolerating 1 failed TaskManagers
> 2016-07-28 22:34:03,302 INFO  org.apache.hadoop.yarn.client.RMProxy
> - Connecting to ResourceManager at node02.mycluster.com/38.111.30.145:8030
> 2016-07-28 22:34:03,324 INFO
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  -
> yarn.client.max-cached-nodemanagers-proxies : 0
> 2016-07-28 22:34:03,325 INFO  org.apache.flink.yarn.YarnJobManager
> - Registering ApplicationMaster with tracking url
> http://node05.mycluster.com:48590.
> 2016-07-28 22:34:03,771 INFO  org.apache.flink.yarn.YarnJobManager
> - Retrieved 0 TaskManagers from previous attempts.
> 2016-07-28 22:34:03,776 INFO  org.apache.flink.yarn.YarnJobManager
> - Requesting initial TaskManager container 0.
> 2016-07-28 22:34:03,811 INFO  org.apache.flink.yarn.Utils
> - Copying from
> file:/mnt/b/hadoop/yarn/local/usercache/user/appcache/application_1467433388200_0136/container_e05_1467433388200_0136_01_000001/flink-conf-modified.yaml
> to
> hdfs://node01.mycluster.com:8020/user/user/.flink/application_1467433388200_0136/flink-conf-modified.yaml
> 2016-07-28 22:34:04,443 INFO  org.apache.flink.yarn.YarnJobManager
> - Prepared local resource for modified yaml: resource { scheme: "hdfs" host:
> "node01.mycluster.com" port: 8020 file:
> "/user/user/.flink/application_1467433388200_0136/flink-conf-modified.yaml"
> } size: 5593 timestamp: 1469745244302 type: FILE visibility: APPLICATION
> 2016-07-28 22:34:04,449 INFO  org.apache.flink.yarn.YarnJobManager
> - Create container launch context.
> 2016-07-28 22:34:04,459 INFO  org.apache.flink.yarn.YarnJobManager
> - Starting TM with command=$JAVA_HOME/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m  -Dlog.file="<LOG_DIR>/taskmanager.log"
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err
> 2016-07-28 22:34:04,463 INFO  org.apache.flink.yarn.YarnJobManager
> - JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager was
> granted leadership with leader session ID
> Some(3d2a8629-081e-44bc-8c38-446b510d5909).
> 2016-07-28 22:34:04,504 INFO  org.apache.flink.yarn.YarnJobManager
> - Delaying recovery of all jobs by 10000 milliseconds.
> 2016-07-28 22:34:04,511 INFO
> org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
> reachable under
> akka.tcp://flink@38.111.30.148:59857/user/jobmanager:3d2a8629-081e-44bc-8c38-446b510d5909.
> 2016-07-28 22:34:04,878 INFO
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Received new
> token for : node03.mycluster.com:45454
> 2016-07-28 22:34:04,900 INFO  org.apache.flink.yarn.YarnJobManager
> - Got new container for allocation:
> container_e05_1467433388200_0136_01_000002
> 2016-07-28 22:34:04,900 INFO  org.apache.flink.yarn.YarnJobManager
> - The user requested 1 containers, 0 running. 1 containers missing
> 2016-07-28 22:34:04,901 INFO  org.apache.flink.yarn.YarnJobManager
> - 1 containers already allocated by YARN. Starting...
> 2016-07-28 22:34:04,902 INFO
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  -
> Opening proxy : node03.mycluster.com:45454
> 2016-07-28 22:34:05,149 INFO  org.apache.flink.yarn.YarnJobManager
> - Launching container (container_e05_1467433388200_0136_01_000002 on host
> node03.mycluster.com).
> 2016-07-28 22:34:07,375 INFO  org.apache.flink.yarn.YarnJobManager
> - Register akka.tcp://flink@38.111.30.144:45229/user/applicationClient as
> client.
> 2016-07-28 22:34:14,524 INFO  org.apache.flink.yarn.YarnJobManager
> - Attempting to recover all jobs.
> 2016-07-28 22:34:14,527 INFO
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - No
> job graph to recover.
> 2016-07-28 22:34:14,528 INFO  org.apache.flink.yarn.YarnJobManager
> - Re-submitting 0 job graphs.
> 2016-07-28 22:34:16,168 INFO
> org.apache.flink.runtime.instance.InstanceManager             - Registered
> TaskManager at node03
> (akka.tcp://flink@38.111.30.146:48700/user/taskmanager) as
> 1d52e723ce440d6181b6e0e98627bfd2. Current number of registered hosts is 1.
> Current number of alive task slots is 1.
> 2016-07-28 22:34:23,098 INFO  org.apache.flink.yarn.YarnJobManager
> - Submitting job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job).
> 2016-07-28 22:34:23,100 INFO  org.apache.flink.yarn.YarnJobManager
> - Using restart strategy NoRestartStrategy for
> 9cdd920b36bec849689a72026402a333.
> 2016-07-28 22:34:23,387 INFO
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - Added
> SubmittedJobGraph(JobGraph(jobId: 9cdd920b36bec849689a72026402a333),
> JobInfo(client: Actor[akka.tcp://flink@127.0.0.1:38212/user/$a#-2098822004]
> (EXECUTION_RESULT_AND_STATE_CHANGES), start: 1469745263098)) to ZooKeeper.
> 2016-07-28 22:34:23,395 INFO  org.apache.flink.yarn.YarnJobManager
> - Scheduling job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job).
> 2016-07-28 22:34:23,397 INFO  org.apache.flink.yarn.YarnJobManager
> - Status of job 9cdd920b36bec849689a72026402a333 (Flink Streaming Job)
> changed to RUNNING.
> 2016-07-28 22:34:23,400 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
> Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282)
> switched from CREATED to SCHEDULED
> 2016-07-28 22:34:23,423 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
> Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282)
> switched from SCHEDULED to DEPLOYING
> 2016-07-28 22:34:23,424 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
> Source: Custom Source -> Sink: Unnamed (1/1) (attempt #0) to node03
> 2016-07-28 22:34:24,554 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
> Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282)
> switched from DEPLOYING to RUNNING
> 2016-07-28 22:34:40,021 WARN  akka.remote.ReliableDeliverySupervisor
> - Association with remote system [akka.tcp://flink@38.111.30.146:48700] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
> - Container container_e05_1467433388200_0136_01_000002 is completed with
> diagnostics: Container
> [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is running
> beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical memory
> used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_e05_1467433388200_0136_01_000002 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
> /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
> -XX:MaxDirectMemorySize=1448m
> -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
> -Dlogback.configurationFile=file:logback.xml
> -Dlog4j.configuration=file:log4j.properties
> org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
> 2>
> /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
> 2016-07-28 22:34:40,400 INFO  org.apache.flink.yarn.YarnJobManager
> - Container container_e05_1467433388200_0136_01_000002 was a running
> container. Total failed containers 1.
> 2016-07-28 22:34:40,404 ERROR org.apache.flink.yarn.YarnJobManager
> - Stopping YARN session because the number of failed containers (1) exceeded
> the maximum failed container count (1). This number is controlled by the
> 'yarn.maximum-failed-containers' configuration setting. By default its the
> number of requested containers
> 2016-07-28 22:34:40,405 INFO  org.apache.flink.yarn.YarnJobManager
> - The user requested 1 containers, 0 running. 1 containers missing
> 2016-07-28 22:34:40,406 INFO  org.apache.flink.yarn.YarnJobManager
> - There are 1 containers missing. 0 are already requested. Requesting 1
> additional container(s) from YARN. Reallocation of failed containers is
> enabled=true ('yarn.reallocate-failed')
> 2016-07-28 22:34:40,408 INFO  org.apache.flink.yarn.YarnJobManager
> - Requested additional container from YARN. Pending requests 1.
> 2016-07-28 22:34:40,410 INFO  org.apache.flink.yarn.YarnJobManager
> - Stopping YARN JobManager with status FAILED and diagnostic Stopping YARN
> session because the number of failed containers (1) exceeded the maximum
> failed container count (1). This number is controlled by the
> 'yarn.maximum-failed-containers' configuration setting. By default its the
> number of requested containers.
> 2016-07-28 22:34:40,419 INFO
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Waiting for
> application to be successfully unregistered.
> 2016-07-28 22:34:40,525 INFO
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl  -
> Interrupted while waiting for queue
> java.lang.InterruptedException
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>     at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287)
> 2016-07-28 22:34:40,528 INFO
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  -
> Opening proxy : node03.mycluster.com:45454
> 2016-07-28 22:34:40,647 INFO  org.apache.flink.yarn.YarnJobManager
> - Stopping JobManager akka.tcp://flink@38.111.30.148:59857/user/jobmanager.
> 2016-07-28 22:34:40,883 INFO
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
> Removed job graph 9cdd920b36bec849689a72026402a333 from ZooKeeper.
> 2016-07-28 22:34:40,911 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
> Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282)
> switched from RUNNING to CANCELING
> 2016-07-28 22:34:40,923 INFO
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
> Stopping ZooKeeperLeaderElectionService.
> 2016-07-28 22:34:40,955 INFO  org.apache.zookeeper.ClientCnxn
> - EventThread shut down
> 2016-07-28 22:34:40,955 INFO  org.apache.zookeeper.ZooKeeper
> - Session: 0x3556f328b2602ff closed
> 2016-07-28 22:34:40,961 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
> Custom Source -> Sink: Unnamed (1/1) (fcae1e4dbfa221cff020dd8140e0b282)
> switched from CANCELING to CANCELED
> 2016-07-28 22:34:41,145 INFO  org.apache.flink.runtime.blob.BlobServer
> - Stopped BLOB server at 0.0.0.0:46326
> 2016-07-28 22:34:41,152 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting
> down remote daemon.
> 2016-07-28 22:34:41,153 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote
> daemon shut down; proceeding with flushing remote transports.
> 2016-07-28 22:34:41,185 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting
> shut down.
> 2016-07-28 22:34:41,199 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  -
> Stopping ZooKeeperLeaderRetrievalService.
> 2016-07-28 22:34:41,202 INFO  org.apache.flink.yarn.YarnJobManager
> - Shutdown completed. Stopping JVM.
> 2016-07-28 22:34:41,208 INFO  org.apache.zookeeper.ZooKeeper
> - Session: 0x15624f916ae08d2 closed
> 2016-07-28 22:34:41,208 INFO  org.apache.zookeeper.ClientCnxn
> - EventThread shut down
> 2016-07-28 22:34:41,209 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
> dashboard root cache directory
> /tmp/flink-web-fbe7069f-8e03-4535-b8d9-71db4f51b298
> 2016-07-28 22:34:41,209 INFO
> org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web
> dashboard jar upload directory
> /tmp/flink-web-upload-1fecf32c-3b86-4d95-ae7b-25def122c3d4
> 2016-07-28 22:34:41,217 INFO
> org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator  - Shutting
> down stack trace sample coordinator.
> End of LogType:jobmanager.log
>
> LogType:jobmanager.out
> Log Upload Time:Thu Jul 28 22:34:43 +0000 2016
> LogLength:0
> Log Contents:
> End of LogType:jobmanager.out
>
>
>
>
> On Fri, Jul 29, 2016 at 11:19 AM, Jack Huang <[hidden email]> wrote:
>>
>> Hi Max,
>>
>> Each events are only a few hundred bytes. I am reading from a Kafka topic
>> from some offset in the past, so the events should be flowing in as fast as
>> Flink can process them.
>>
>> The entire YARN task log, which contains both JobManager and TaskManager
>> outputs, is attached.
>>
>> Thanks a lot,
>> Jack
>>
>>
>> On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]>
>> wrote:
>>>
>>> Hi Jack,
>>>
>>> Considering the type of job you're running, you shouldn't run out of
>>> memory. Could it be that the events are quite large strings? It could
>>> be that the TextOutputFormat doesn't write to disk fast enough and
>>> accumulates memory. Actually, it doesn't perform regular flushing
>>> which could be an issue.
>>>
>>> I'm just guessing, we need to investigate further. Could you please
>>> supply the entire JobManager log file output?
>>>
>>> Thanks,
>>> Max
>>>
>>> On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
>>> > Hi all,
>>> >
>>> > I am running a test Flink streaming task under YARN. It reads messages
>>> > from
>>> > a Kafka topic and writes them to local file system.
>>> >
>>> > object PricerEvent {
>>> >     def main(args:Array[String]) {
>>> >         val kafkaProp = new Properties()
>>> >         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>>> >         kafkaProp.setProperty("auto.offset.reset", "earliest")
>>> >
>>> >         val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> >         env.setStateBackend(new MemoryStateBackend)
>>> >
>>> >         val wins = env.addSource(new
>>> > FlinkKafkaConsumer09[String]("events",
>>> > new SimpleStringSchema, kafkaProp))
>>> >         wins.writeAsText("/home/user/flink_out/" + new
>>> > SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>>> >
>>> >         env.execute
>>> >     }
>>> > }
>>> >
>>> > With the command
>>> >
>>> > flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
>>> > /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>>> >
>>> >
>>> > The task runs fine for a moment and then terminates. I looked into the
>>> > error
>>> > log and found following out-of-memory error message:
>>> >
>>> > 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
>>> > - Container container_e05_1467433388200_0136_01_000002 is completed
>>> > with
>>> > diagnostics: Container
>>> > [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is
>>> > running
>>> > beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical
>>> > memory
>>> > used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
>>> > Dump of the process-tree for container_e05_1467433388200_0136_01_000002
>>> > :
>>> >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> >     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>>> >     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
>>> > 2>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>>> >
>>> > Container killed on request. Exit code is 143
>>> > Container exited with a non-zero exit code 143
>>> >
>>> >
>>> > I don't understand how it should fail due to out-of-memory error. I
>>> > would
>>> > expect the task to slow down if there are too many messages to process,
>>> > but
>>> > not failing altogether. I am not storing any states either.
>>> >
>>> > Does anyone know the reason and the way to fix/avoid this issue?
>>> >
>>> >
>>> > Thanks,
>>> > Jack
>>> >
>>> >
>>> >
>>> >
>>
>>
>
Jack Huang-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Jack Huang-2
Hi Max,

Is there a way to limit the JVM memory usage (something like the -Xmx flag) for the task manager so that it won't go over the YARN limit but will just run GC until there is memory to use? Trying to allocate "enough" memory for this stream task is not ideal because I could have indefinitely many messages backed-up in the source to be process.

Thanks,
Jack


On Tue, Aug 2, 2016 at 5:21 AM, Maximilian Michels <[hidden email]> wrote:
Your job creates a lot of String objects which need to be garbage
collected. It could be that the JVM is not fast enough and Yarn kills
the JVM for consuming too much memory.

You can try two things:

1) Give the task manager more memory
2) Increase the Yarn heap cutoff ratio (e.g yarn.heap-cutoff-ratio: 0.4)

If the error still occurs then we need to investigate further.

Thanks,
Max


>
>
>
>
> On Fri, Jul 29, 2016 at 11:19 AM, Jack Huang <[hidden email]> wrote:
>>
>> Hi Max,
>>
>> Each events are only a few hundred bytes. I am reading from a Kafka topic
>> from some offset in the past, so the events should be flowing in as fast as
>> Flink can process them.
>>
>> The entire YARN task log, which contains both JobManager and TaskManager
>> outputs, is attached.
>>
>> Thanks a lot,
>> Jack
>>
>>
>> On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]>
>> wrote:
>>>
>>> Hi Jack,
>>>
>>> Considering the type of job you're running, you shouldn't run out of
>>> memory. Could it be that the events are quite large strings? It could
>>> be that the TextOutputFormat doesn't write to disk fast enough and
>>> accumulates memory. Actually, it doesn't perform regular flushing
>>> which could be an issue.
>>>
>>> I'm just guessing, we need to investigate further. Could you please
>>> supply the entire JobManager log file output?
>>>
>>> Thanks,
>>> Max
>>>
>>> On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
>>> > Hi all,
>>> >
>>> > I am running a test Flink streaming task under YARN. It reads messages
>>> > from
>>> > a Kafka topic and writes them to local file system.
>>> >
>>> > object PricerEvent {
>>> >     def main(args:Array[String]) {
>>> >         val kafkaProp = new Properties()
>>> >         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>>> >         kafkaProp.setProperty("auto.offset.reset", "earliest")
>>> >
>>> >         val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> >         env.setStateBackend(new MemoryStateBackend)
>>> >
>>> >         val wins = env.addSource(new
>>> > FlinkKafkaConsumer09[String]("events",
>>> > new SimpleStringSchema, kafkaProp))
>>> >         wins.writeAsText("/home/user/flink_out/" + new
>>> > SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>>> >
>>> >         env.execute
>>> >     }
>>> > }
>>> >
>>> > With the command
>>> >
>>> > flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
>>> > /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>>> >
>>> >
>>> > The task runs fine for a moment and then terminates. I looked into the
>>> > error
>>> > log and found following out-of-memory error message:
>>> >
>>> > 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
>>> > - Container container_e05_1467433388200_0136_01_000002 is completed
>>> > with
>>> > diagnostics: Container
>>> > [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is
>>> > running
>>> > beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical
>>> > memory
>>> > used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
>>> > Dump of the process-tree for container_e05_1467433388200_0136_01_000002
>>> > :
>>> >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> >     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>>> >     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
>>> > 2>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>>> >
>>> > Container killed on request. Exit code is 143
>>> > Container exited with a non-zero exit code 143
>>> >
>>> >
>>> > I don't understand how it should fail due to out-of-memory error. I
>>> > would
>>> > expect the task to slow down if there are too many messages to process,
>>> > but
>>> > not failing altogether. I am not storing any states either.
>>> >
>>> > Does anyone know the reason and the way to fix/avoid this issue?
>>> >
>>> >
>>> > Thanks,
>>> > Jack
>>> >
>>> >
>>> >
>>> >
>>
>>
>
Jack Huang-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Jack Huang-2
Hi Max,

Changing yarn-heap-cutoff-ratio works seem to suffice for the time being. Thanks for your help.

Regards,
Jack

On Tue, Aug 2, 2016 at 11:11 AM, Jack Huang <[hidden email]> wrote:
Hi Max,

Is there a way to limit the JVM memory usage (something like the -Xmx flag) for the task manager so that it won't go over the YARN limit but will just run GC until there is memory to use? Trying to allocate "enough" memory for this stream task is not ideal because I could have indefinitely many messages backed-up in the source to be process.

Thanks,
Jack


On Tue, Aug 2, 2016 at 5:21 AM, Maximilian Michels <[hidden email]> wrote:
Your job creates a lot of String objects which need to be garbage
collected. It could be that the JVM is not fast enough and Yarn kills
the JVM for consuming too much memory.

You can try two things:

1) Give the task manager more memory
2) Increase the Yarn heap cutoff ratio (e.g yarn.heap-cutoff-ratio: 0.4)

If the error still occurs then we need to investigate further.

Thanks,
Max


>
>
>
>
 > On Fri, Jul 29, 2016 at 11:19 AM, Jack Huang <[hidden email]> wrote:
>>
>> Hi Max,
>>
>> Each events are only a few hundred bytes. I am reading from a Kafka topic
>> from some offset in the past, so the events should be flowing in as fast as
>> Flink can process them.
>>
>> The entire YARN task log, which contains both JobManager and TaskManager
>> outputs, is attached.
>>
>> Thanks a lot,
>> Jack
>>
>>
>> On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]>
>> wrote:
>>>
>>> Hi Jack,
>>>
>>> Considering the type of job you're running, you shouldn't run out of
>>> memory. Could it be that the events are quite large strings? It could
>>> be that the TextOutputFormat doesn't write to disk fast enough and
>>> accumulates memory. Actually, it doesn't perform regular flushing
>>> which could be an issue.
>>>
>>> I'm just guessing, we need to investigate further. Could you please
>>> supply the entire JobManager log file output?
>>>
>>> Thanks,
>>> Max
>>>
>>> On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
>>> > Hi all,
>>> >
>>> > I am running a test Flink streaming task under YARN. It reads messages
>>> > from
>>> > a Kafka topic and writes them to local file system.
>>> >
>>> > object PricerEvent {
>>> >     def main(args:Array[String]) {
>>> >         val kafkaProp = new Properties()
>>> >         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>>> >         kafkaProp.setProperty("auto.offset.reset", "earliest")
>>> >
>>> >         val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> >         env.setStateBackend(new MemoryStateBackend)
>>> >
>>> >         val wins = env.addSource(new
>>> > FlinkKafkaConsumer09[String]("events",
>>> > new SimpleStringSchema, kafkaProp))
>>> >         wins.writeAsText("/home/user/flink_out/" + new
>>> > SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>>> >
>>> >         env.execute
>>> >     }
>>> > }
>>> >
>>> > With the command
>>> >
>>> > flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
>>> > /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>>> >
>>> >
>>> > The task runs fine for a moment and then terminates. I looked into the
>>> > error
>>> > log and found following out-of-memory error message:
>>> >
>>> > 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
>>> > - Container container_e05_1467433388200_0136_01_000002 is completed
>>> > with
>>> > diagnostics: Container
>>> > [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is
>>> > running
>>> > beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical
>>> > memory
>>> > used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
>>> > Dump of the process-tree for container_e05_1467433388200_0136_01_000002
>>> > :
>>> >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> >     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>>> >     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
>>> > 2>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>>> >
>>> > Container killed on request. Exit code is 143
>>> > Container exited with a non-zero exit code 143
>>> >
>>> >
>>> > I don't understand how it should fail due to out-of-memory error. I
>>> > would
>>> > expect the task to slow down if there are too many messages to process,
>>> > but
>>> > not failing altogether. I am not storing any states either.
>>> >
>>> > Does anyone know the reason and the way to fix/avoid this issue?
>>> >
>>> >
>>> > Thanks,
>>> > Jack
>>> >
>>> >
>>> >
>>> >
>>
>>
>
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Container running beyond physical memory limits when processing DataStream

Stephan Ewen
Hi!

The JVM is allowed 1448m of memory, and the JVM should never use more heap than that.

The fact that the process is using more than 2GB memory in total means that some libraries are allocating memory outside the heap.


Greetings,
Stephan


On Thu, Aug 4, 2016 at 3:41 AM, Jack Huang <[hidden email]> wrote:
Hi Max,

Changing yarn-heap-cutoff-ratio works seem to suffice for the time being. Thanks for your help.

Regards,
Jack

On Tue, Aug 2, 2016 at 11:11 AM, Jack Huang <[hidden email]> wrote:
Hi Max,

Is there a way to limit the JVM memory usage (something like the -Xmx flag) for the task manager so that it won't go over the YARN limit but will just run GC until there is memory to use? Trying to allocate "enough" memory for this stream task is not ideal because I could have indefinitely many messages backed-up in the source to be process.

Thanks,
Jack


On Tue, Aug 2, 2016 at 5:21 AM, Maximilian Michels <[hidden email]> wrote:
Your job creates a lot of String objects which need to be garbage
collected. It could be that the JVM is not fast enough and Yarn kills
the JVM for consuming too much memory.

You can try two things:

1) Give the task manager more memory
2) Increase the Yarn heap cutoff ratio (e.g yarn.heap-cutoff-ratio: 0.4)

If the error still occurs then we need to investigate further.

Thanks,
Max


>
>
>
>
 > On Fri, Jul 29, 2016 at 11:19 AM, Jack Huang <[hidden email]> wrote:
>>
>> Hi Max,
>>
>> Each events are only a few hundred bytes. I am reading from a Kafka topic
>> from some offset in the past, so the events should be flowing in as fast as
>> Flink can process them.
>>
>> The entire YARN task log, which contains both JobManager and TaskManager
>> outputs, is attached.
>>
>> Thanks a lot,
>> Jack
>>
>>
>> On Fri, Jul 29, 2016 at 2:04 AM, Maximilian Michels <[hidden email]>
>> wrote:
>>>
>>> Hi Jack,
>>>
>>> Considering the type of job you're running, you shouldn't run out of
>>> memory. Could it be that the events are quite large strings? It could
>>> be that the TextOutputFormat doesn't write to disk fast enough and
>>> accumulates memory. Actually, it doesn't perform regular flushing
>>> which could be an issue.
>>>
>>> I'm just guessing, we need to investigate further. Could you please
>>> supply the entire JobManager log file output?
>>>
>>> Thanks,
>>> Max
>>>
>>> On Fri, Jul 29, 2016 at 12:59 AM, Jack Huang <[hidden email]> wrote:
>>> > Hi all,
>>> >
>>> > I am running a test Flink streaming task under YARN. It reads messages
>>> > from
>>> > a Kafka topic and writes them to local file system.
>>> >
>>> > object PricerEvent {
>>> >     def main(args:Array[String]) {
>>> >         val kafkaProp = new Properties()
>>> >         kafkaProp.setProperty("bootstrap.servers", "localhost:6667")
>>> >         kafkaProp.setProperty("auto.offset.reset", "earliest")
>>> >
>>> >         val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> >         env.setStateBackend(new MemoryStateBackend)
>>> >
>>> >         val wins = env.addSource(new
>>> > FlinkKafkaConsumer09[String]("events",
>>> > new SimpleStringSchema, kafkaProp))
>>> >         wins.writeAsText("/home/user/flink_out/" + new
>>> > SimpleDateFormat("yyyy-MM-dd_HH-mm-ss").format(new Date))
>>> >
>>> >         env.execute
>>> >     }
>>> > }
>>> >
>>> > With the command
>>> >
>>> > flink run -m yarn-cluster -yn 1 -ytm 2048 -c PricerEvent
>>> > /home/user/flink-example/build/libs/flink-example-1.0-all.jar
>>> >
>>> >
>>> > The task runs fine for a moment and then terminates. I looked into the
>>> > error
>>> > log and found following out-of-memory error message:
>>> >
>>> > 2016-07-28 22:34:40,397 INFO  org.apache.flink.yarn.YarnJobManager
>>> > - Container container_e05_1467433388200_0136_01_000002 is completed
>>> > with
>>> > diagnostics: Container
>>> > [pid=5832,containerID=container_e05_1467433388200_0136_01_000002] is
>>> > running
>>> > beyond physical memory limits. Current usage: 2.3 GB of 2 GB physical
>>> > memory
>>> > used; 6.1 GB of 4.2 GB virtual memory used. Killing container.
>>> > Dump of the process-tree for container_e05_1467433388200_0136_01_000002
>>> > :
>>> >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>>> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> >     |- 5838 5832 5832 5832 (java) 2817 481 6553391104 592779
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
>>> >     |- 5832 5830 5832 5832 (bash) 0 0 12759040 357 /bin/bash -c
>>> > /usr/jdk64/jdk1.8.0_60/bin/java -Xms1448m -Xmx1448m
>>> > -XX:MaxDirectMemorySize=1448m
>>> >
>>> > -Dlog.file=/mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.log
>>> > -Dlogback.configurationFile=file:logback.xml
>>> > -Dlog4j.configuration=file:log4j.properties
>>> > org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.out
>>> > 2>
>>> >
>>> > /mnt/a/hadoop/yarn/log/application_1467433388200_0136/container_e05_1467433388200_0136_01_000002/taskmanager.err
>>> >
>>> > Container killed on request. Exit code is 143
>>> > Container exited with a non-zero exit code 143
>>> >
>>> >
>>> > I don't understand how it should fail due to out-of-memory error. I
>>> > would
>>> > expect the task to slow down if there are too many messages to process,
>>> > but
>>> > not failing altogether. I am not storing any states either.
>>> >
>>> > Does anyone know the reason and the way to fix/avoid this issue?
>>> >
>>> >
>>> > Thanks,
>>> > Jack
>>> >
>>> >
>>> >
>>> >
>>
>>
>
Free forum by Nabble Edit this page