Cannot get Python API to work at all; java.io.FileNotFoundException for Flink python resources

[Ronkainen Jussi]

Hi,

 

I’m trying to evaluate the Python API but get a java.io.FileNotFoundException when trying to run any Python script on pyflink. The contents of the Python script are irrelevant, doesn’t even need to be valid Python.

I made a local install of 0.9.1, copy-pasted the example Python script from https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/python.html to a file “~/python_src/wordcount.py” and tried to run it in Flink root directory (/home/flink/flink-0.9.1/):

 

./bin/pyflink2.sh ~/python_src/wordcount.py

 

I get a java.io.FileNotFoundException (trace below). The cause could be related to PythonPlanBinder line 93, which strips 7 characters from FLINK_DIR and after that appends /resources/python to the result, ending in an invalid path. If I understand correctly, for my  user “flink” the path should be /home/flink/flink-0.9.1/resources/python

 

Am I missing some setup option, using pyflink wrong, or could this be a bug? I also tried compiling from 0.10-SNAPSHOT but the problem persists.

I’m running Ubuntu 14.04  with openjdk 8 and Python 2.7.6.

 

Exception trace:

 

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.

        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)

        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)

        at org.apache.flink.client.program.Client.run(Client.java:315)

        at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:582)

        at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)

        at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:878)

        at org.apache.flink.client.CliFrontend.main(CliFrontend.java:920)

Caused by: java.io.FileNotFoundException: File /home/flink/flin/resources/python does not exist or the user running Flink ('flink') has insufficient permissions to access it.

        at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:107)

        at org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:242)

        at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.prepareFiles(PythonPlanBinder.java:138)

        at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:108)

        at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)

        ... 6 more

 

-        Jussi

[Maximilian Michels]

Hi Ronkainen,

Sorry for the late reply. Unfortunately, this is a bug in the Python
API. I've reproduced the issue and fixed it for the upcoming releases.
The fix will be included in the 0.9.2 and the 0.10 release. If you
don't mind, you could already use the 0.10-SNAPSHOT version (0.10
release will be out next week). Also, the bug is not present in Flink
0.9.0.

Thanks,
Max

On Mon, Oct 19, 2015 at 1:27 PM, Ronkainen Jussi <[hidden email]> wrote:

> Hi,
>
>
>
> I’m trying to evaluate the Python API but get a
> java.io.FileNotFoundException when trying to run any Python script on
> pyflink. The contents of the Python script are irrelevant, doesn’t even need
> to be valid Python.
>
> I made a local install of 0.9.1, copy-pasted the example Python script from
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/python.html
> to a file “~/python_src/wordcount.py” and tried to run it in Flink root
> directory (/home/flink/flink-0.9.1/):
>
>
>
> ./bin/pyflink2.sh ~/python_src/wordcount.py
>
>
>
> I get a java.io.FileNotFoundException (trace below). The cause could be
> related to PythonPlanBinder line 93, which strips 7 characters from
> FLINK_DIR and after that appends /resources/python to the result, ending in
> an invalid path. If I understand correctly, for my  user “flink” the path
> should be /home/flink/flink-0.9.1/resources/python
>
>
>
> Am I missing some setup option, using pyflink wrong, or could this be a bug?
> I also tried compiling from 0.10-SNAPSHOT but the problem persists.
>
> I’m running Ubuntu 14.04  with openjdk 8 and Python 2.7.6.
>
>
>
> Exception trace:
>
>
>
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
>
>         at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
>
>         at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>
>         at org.apache.flink.client.program.Client.run(Client.java:315)
>
>         at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:582)
>
>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>
>         at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:878)
>
>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:920)
>
> Caused by: java.io.FileNotFoundException: File
> /home/flink/flin/resources/python does not exist or the user running Flink
> ('flink') has insufficient permissions to access it.
>
>         at
> org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:107)
>
>         at
> org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:242)
>
>         at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.prepareFiles(PythonPlanBinder.java:138)
>
>         at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:108)
>
>         at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:497)
>
>         at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>
>         ... 6 more
>
>
>
> -        Jussi