PyFlink: Shall we disallow relative URL for filesystem path?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

PyFlink: Shall we disallow relative URL for filesystem path?

Yik San Chan
Hi community,

When using Filesystem SQL Connector, users need to provide a path. When running a PyFlink job using the mini-cluster mode by simply `python WordCount.py`, the path can be a relative path, such as, `words.txt`. However, trying to submit the job to `flink run` will fail without question, because `flink run` expects an absolute file path prefix-ed with a `file://`, for example, `file:///tmp/input/words.txt`.

Since we will need to submit our job implementation to run on prod anyway, I recommend disallowing relative paths such as `words.txt`. This will save users from spending hours to figure out "Why I can `python` run this job, while `flink run` fails?"

Look forward to your feedback.

Best,
Yik San