Jython support for Flink

[Dante Van den Broeke]

Dear,

I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.

The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.

I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online. 

thanks in advance.

kind regards,

Dante Van den Broeke

[Jeff Zhang]

Hi Dante,

Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html

Dante Van den Broeke <[hidden email]> 于2019年7月19日周五 下午10:40写道：

Dear,

I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.

The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.

I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online. 

thanks in advance.

kind regards,

Dante Van den Broeke

--

[tison]

Hi Dante,

Both Jython and Jython support for Flink are out of development

and maintain. As pointed out by Jeff, Flink 1.9 supports Python

api via py4j[1] and the document page as posted.

I guess your algorithms are written in CPython instead of Jython

and want Jython only for interoperate, and thus recommend you to

have a look at the doc posted above.

For previous Jython support example or setup, cc Chesnay who is

our committer familiar with this scope.

[1] https://www.py4j.org/

Jeff Zhang <[hidden email]> 于2019年7月19日周五 下午11:06写道：

Hi Dante,

Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html

Dante Van den Broeke <[hidden email]> 于2019年7月19日周五 下午10:40写道：

Dear,

I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.

The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.

I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online. 

thanks in advance.

kind regards,

Dante Van den Broeke

--

[Dante Van den Broeke]

Hi Tison, Jeff,

Thanks a lot for the help! I’ll definitly look into the python API and py4j support next. I was also thinking about trying to create the pipeline through beam and flink instead of kafka and flink, as i see that python is a full class citizen in the beam framework! 

Regards,

Dante

Op 20 jul. 2019 om 01:59 heeft Zili Chen <[hidden email]> het volgende geschreven:

Hi Dante,

Both Jython and Jython support for Flink are out of development

and maintain. As pointed out by Jeff, Flink 1.9 supports Python

api via py4j[1] and the document page as posted.

I guess your algorithms are written in CPython instead of Jython

and want Jython only for interoperate, and thus recommend you to

have a look at the doc posted above.

For previous Jython support example or setup, cc Chesnay who is

our committer familiar with this scope.

[1] https://www.py4j.org/

Jeff Zhang <[hidden email]> 于2019年7月19日周五 下午11:06写道：

Hi Dante,

Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html

Dante Van den Broeke <[hidden email]> 于2019年7月19日周五 下午10:40写道：

Dear,

I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.

The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.

I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online. 

thanks in advance.

kind regards,

Dante Van den Broeke

--

[tison]

Hi Dante,

Nice finding! I just miss this powerful project :-)

Dante Van den Broeke <[hidden email]> 于2019年7月20日周六 下午5:28写道：

Hi Tison, Jeff,

Thanks a lot for the help! I’ll definitly look into the python API and py4j support next. I was also thinking about trying to create the pipeline through beam and flink instead of kafka and flink, as i see that python is a full class citizen in the beam framework! 

Regards,

Dante

Op 20 jul. 2019 om 01:59 heeft Zili Chen <[hidden email]> het volgende geschreven:

Hi Dante,

Both Jython and Jython support for Flink are out of development

and maintain. As pointed out by Jeff, Flink 1.9 supports Python

api via py4j[1] and the document page as posted.

I guess your algorithms are written in CPython instead of Jython

and want Jython only for interoperate, and thus recommend you to

have a look at the doc posted above.

For previous Jython support example or setup, cc Chesnay who is

our committer familiar with this scope.

[1] https://www.py4j.org/

Jeff Zhang <[hidden email]> 于2019年7月19日周五 下午11:06写道：

Hi Dante,

Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html

Dante Van den Broeke <[hidden email]> 于2019年7月19日周五 下午10:40写道：

Dear,

I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.

The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.

I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online. 

thanks in advance.

kind regards,

Dante Van den Broeke

--