Flink program without a line of code

[Alexander Smirnov]

Hi guys!

I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!

The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.

Is there a similar functionality in Flink? May be some third party plugin?

Thank you,
Alex

[Fabian Hueske-2]

Hi Alex,

welcome to the Flink community!

Right now, there is no way to specify a Flink program without writing code (Java, Scala, Python(beta)).

In principle it is possible to put such functionality on top of the DataStream or DataSet APIs.
This has been done before for other programming APIs (Flink's own libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google DataFlow, Mahout, Cascading, ...). However, all of these are again programming APIs, some specialized for certain use-cases.

Specifying Flink programs by config files (or graphically) would require a data model, a DataStream/DataSet program generator and probably a code generation component.

Best, Fabian

2016-04-22 18:41 GMT+02:00 Alexander Smirnov <[hidden email]>:

Hi guys!

I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!

The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.

Is there a similar functionality in Flink? May be some third party plugin?

Thank you,
Alex

[Aljoscha Krettek]

Hi,

I think if the Table API/SQL API evolves enough it should be able to supply a Flink program as just an SQL query with source/sink definitions. Hopefully, in the future. :-)

Cheers,

Aljoscha

On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <[hidden email]> wrote:

Hi Alex,

welcome to the Flink community!

Right now, there is no way to specify a Flink program without writing code (Java, Scala, Python(beta)).

In principle it is possible to put such functionality on top of the DataStream or DataSet APIs.
This has been done before for other programming APIs (Flink's own libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google DataFlow, Mahout, Cascading, ...). However, all of these are again programming APIs, some specialized for certain use-cases.

Specifying Flink programs by config files (or graphically) would require a data model, a DataStream/DataSet program generator and probably a code generation component.

Best, Fabian

2016-04-22 18:41 GMT+02:00 Alexander Smirnov <[hidden email]>:

Hi guys!

I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!

The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.

Is there a similar functionality in Flink? May be some third party plugin?

Thank you,
Alex

[Flavio Pompermaier]

Hi Alexander,
since I was looking for something similar some days ago here is what I know about this argument:
during the Stratosphere project there was Meteor and Supremo allowing that [1] but then it was dismissed in favour of Pig integration that I don't wheter it was ever completed. Yiu could give a try to Piglet project[2] that allows to use PIG with Spark and Flink but I don't know how well it works (Flink integration is also very recent and not documented anywhere).

Best,
Flavio

[1] http://stratosphere.eu/assets/papers/Sopremo_Meteor%20BigData.pdf
[2] https://github.com/ksattler/piglet

On 23 Apr 2016 07:48, "Aljoscha Krettek" <[hidden email]> wrote:

Hi,

I think if the Table API/SQL API evolves enough it should be able to supply a Flink program as just an SQL query with source/sink definitions. Hopefully, in the future. :-)

Cheers,

Aljoscha

On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <[hidden email]> wrote:

Hi Alex,

welcome to the Flink community!

Right now, there is no way to specify a Flink program without writing code (Java, Scala, Python(beta)).

In principle it is possible to put such functionality on top of the DataStream or DataSet APIs.
This has been done before for other programming APIs (Flink's own libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google DataFlow, Mahout, Cascading, ...). However, all of these are again programming APIs, some specialized for certain use-cases.

Specifying Flink programs by config files (or graphically) would require a data model, a DataStream/DataSet program generator and probably a code generation component.

Best, Fabian

2016-04-22 18:41 GMT+02:00 Alexander Smirnov <[hidden email]>:

Hi guys!

I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!

The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.

Is there a similar functionality in Flink? May be some third party plugin?

Thank you,
Alex

[Alexander Smirnov]

thank you so much for the responses, guys!

On Sat, Apr 23, 2016 at 12:09 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Alexander,
since I was looking for something similar some days ago here is what I know about this argument:
during the Stratosphere project there was Meteor and Supremo allowing that [1] but then it was dismissed in favour of Pig integration that I don't wheter it was ever completed. Yiu could give a try to Piglet project[2] that allows to use PIG with Spark and Flink but I don't know how well it works (Flink integration is also very recent and not documented anywhere).

Best,
Flavio

[1] http://stratosphere.eu/assets/papers/Sopremo_Meteor%20BigData.pdf
[2] https://github.com/ksattler/piglet

On 23 Apr 2016 07:48, "Aljoscha Krettek" <[hidden email]> wrote:

Hi,

I think if the Table API/SQL API evolves enough it should be able to supply a Flink program as just an SQL query with source/sink definitions. Hopefully, in the future. :-)

Cheers,

Aljoscha

On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <[hidden email]> wrote:

Hi Alex,

welcome to the Flink community!

Right now, there is no way to specify a Flink program without writing code (Java, Scala, Python(beta)).

In principle it is possible to put such functionality on top of the DataStream or DataSet APIs.
This has been done before for other programming APIs (Flink's own libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google DataFlow, Mahout, Cascading, ...). However, all of these are again programming APIs, some specialized for certain use-cases.

Specifying Flink programs by config files (or graphically) would require a data model, a DataStream/DataSet program generator and probably a code generation component.

Best, Fabian

2016-04-22 18:41 GMT+02:00 Alexander Smirnov <[hidden email]>:

Hi guys!

I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!

The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.

Is there a similar functionality in Flink? May be some third party plugin?

Thank you,
Alex