flink-job-in-yarn,has max memory
Locked
 
|
Hi, i have a flink job,and abt assembly to get a jar file,so i put it to yarn and run it,use the follow commend: ------------------------------------------------------------------------ /home/www/flink-1.1.1/bin/flink run \ -m yarn-cluster \ -yn 1 \ -ys 2 \ -yjm 4096 \ -ytm 4096 \ --class skRecomm.SkProRecommFlink \ --classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar \ --classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar \ --classpath file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar \ --classpath file:///opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar \ --classpath file:///opt/cloudera/parcels/CDH/lib/hbase/lib/guava-12.0.1.jar \ /home/www/flink-mining/deploy/zx_article-7cffb87.jar -----------------------------------------------------------------------------------the commend is in supervisor on a computer(*,*,*,22), ---------------------------- and flink/conf/flink-conf.yaml,i set those pargam, ------------------------------------------ fs.hdfs.hadoopconf: /etc/hadoop/conf/ jobmanager.web.port: 8081 parallelism.default: 1 taskmanager.memory.preallocate: false taskmanager.numberOfTaskSlots: 1 taskmanager.heap.mb: 512 jobmanager.heap.mb: 256 arallelism.default: 1 jobmanager.rpc.port: 6123 jobmanager.rpc.address: localhost
------------------------------------------ the job is success, can find follow message in yarn monitor, flink.base.dir.path /data1/yarn/nm/usercache/work/appcache/application_1472623395420_36719/container_e03_1472623395420_36719_01_000001 fs.hdfs.hadoopconf /etc/hadoop/conf/ jobmanager.heap.mb 256 jobmanager.rpc.address *.*.*.79 -----(is not *.*.*.22,and taskmanager is *.*.*.69) jobmanager.rpc.port 32987 jobmanager.web.port 0 parallelism.default 1 recovery.zookeeper.path.namespace application_1472623395420_36719 taskmanager.heap.mb 512 taskmanager.memory.preallocate false taskmanager.numberOfTaskSlots 1 ----------------------------------------------------- Overview Data Port All Slots Free Slots CPU Cores Physical Memory Free Memory Flink Managed Memory 30471 2 0 32 189 GB 2.88 GB 1.96 GB ----------------------------------------------------------------------------------------------------------------------- Memory JVM (Heap/Non-Heap) Type Committed Initial Maximum Heap 2.92 GB 3.00 GB 2.92 GB Non-Heap 53.4 MB 23.4 MB 130 MB Total 2.97 GB 3.02 GB 3.04 GB ----------------------------------------------------------------- Outside JVM Type Count Used Capacity Direct 510 860 KB 860 KB Mapped 0 0 B 0 B ------------------------------------------------------------------- i find in computer(*,*,*,22),the pid=345 has 2.36g memory,and the pid=345 is the job that from supervisor run, i really do not know why ?the job was run in yarn ,why occupy so much memory in computer(*.*.*.22),i just run the job in computer(*.*.*.22). thank you answer my question. |
|
|
Hi, The TaskManager reports a total memory usage of 3 GB. That's fine, given that you requested containers of size 4GB. Flink doesn't allocate all the memory assigned to the container to the heap. Are you running a batch or a streaming job? On Tue, Nov 29, 2016 at 12:43 PM, <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |