Re: Pyflink 1.10.0 issue on cluster

Posted by Xingbo Huang on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Pyflink-1-10-0-issue-on-cluster-tp36959p36961.html

Hi rookieCOder,
You need to make sure that your files can be read by each slaves, so an alternative solution is to put your files on hdfs

Best,
Xingbo

rookieCOder <[hidden email]> 于2020年7月27日周一 下午5:49写道：

'm coding with pyflink 1.10.0 and building cluster with flink 1.10.0
I define the source and the sink as following:
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t2674/%E6%97%A0%E6%A0%87%E9%A2%98.png>
When I run this code only on master, it's OK. When I run this code on
cluster, with 1 master and 1 salve, and I submit the task on master like
this:
sudo flink-1.10.0/bin/flink run -py main.py
And error occurs like:
Caused by: java.io.FileNotFoundException: The provided file path
/opt/raw_data/input_json_65adbe54-cfdc-11ea-9d47-020012970011 does not
exist.
This file is stored on master's local file system. It seems that the slaves
read
their own file system instead of the master's. Or maybe there are other
reasons.
The question is how can I avoid the error?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/