Flink 1.12.1 example applications failing on a single node yarn cluster
Locked
 
|
I am trying out flink example as explained in flink docs in a single node yarn cluster.
On executing ubuntu@vrni-platform:~/build-target/flink$ ./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar It is failing with the below errors I have made the log level DEBUG and I do see that
The entire DEBUG logs are placed here. Nodemanager logs are placed here. Can someone let me know what is going wrong? Does flink not support single node yarn cluster for development? |
|
|
The same has been asked in StackOverflow also. Any suggestions here? On Wed, Feb 24, 2021 at 10:25 PM Debraj Manna <[hidden email]> wrote:
|
|
|
In my setup hadoop-yarn-nodemenager is running with yarn user. ubuntu@vrni-platform:/tmp/flink$ ps -ef | grep nodemanager yarn 4953 1 2 05:53 ? 00:11:26 /usr/lib/jvm/java-8-openjdk/bin/java -Dproc_nodemanager -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/heap-dumps/yarn -XX:+ExitOnOutOfMemoryError -Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-yarn-nodemanager-vrni-platform.log -Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/usr/lib/hadoop/lib/native -Xmx512m -Dhadoop.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=hadoop-yarn-nodemanager-vrni-platform.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=yarn -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.yarn.server.nodemanager.NodeManager I was executing the ./bin/flink command as ubuntu user and yarn user does not have permission to write to ubuntu's home folder in my setup. ubuntu@vrni-platform:/tmp/flink$ echo ~ubuntu /home/ubuntu ubuntu@vrni-platform:/tmp/flink$ echo ~yarn /var/lib/hadoop-yarn It appears to me flink needs permission to write to user's home directory to create a .flink folder even when the job is submitted in yarn. It is working fine for me if I run the flink with yarn user. in my setup. Just for my knowledge is there any config in flink to specify the location of .flink folder? On Thu, Feb 25, 2021 at 10:48 AM Debraj Manna <[hidden email]> wrote:
|
|
|
Hi Debraj, thanks for reaching out to the Flink community. Without knowing the details on how you've set up the Single-Node YARN cluster, I would still guess that it is a configuration issue on the YARN side. Flink does not know about a .flink folder. Hence, there is no configuration to set this folder. Best, Matthias On Fri, Feb 26, 2021 at 2:40 PM Debraj Manna <[hidden email]> wrote:
|
|
|
Thanks Matthias for replying. Yes there was some yarn configuration issue on my side which I mentioned in my last email. I am starting on flink. So just for my understanding in few links (posted below) it is reported that flink needs to create a .flink directory in the users home folder. Even though I am not using HDFS with yarn (in single-node deployment) but I am also observing the same. Is there a way I can configure the location where flink stores the jar and configuration file as mentioned in the below link? From the above link "Flink creates a Same mentioned here. On Fri, Feb 26, 2021 at 9:45 PM Matthias Pohl <[hidden email]> wrote:
|
|
|
Hi Debrai, sorry for misleading you first. You're right. I looked through the code once more and found something: There's the yarn.staging-directory [1] that is set to the user's home folder by default. This parameter is used by the YarnApplicationFileUploader [2] to upload the application files. I hope that helps. Best, Matthias On Fri, Feb 26, 2021 at 5:51 PM Debraj Manna <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |