Efficient datatypes?

[Kruse, Sebastian]

Hi everyone,

 

I think that during one of the meetups, it was mentioned that Flink can in some cases operate on serialized data. Given I understood that correctly, which cases that would be, i.e, which data types and operators support such a feature?

 

Cheers,

Sebastian

 

 

---

Sebastian Kruse

Doktorand am Fachbereich Information Systems Group

Hasso-Plattner-Institut an der Universität Potsdam
Prof.-Dr.-Helmert-Str. 2-3, D-14482 Potsdam
Tel [phone]

Amtsgericht Potsdam, HRB 12184
Geschäftsführung: Prof. Dr. Christoph Meinel

 

 

[Stephan Ewen]

Hey!

All data types are always kept serialized for caching/hashing/sorting. Deserialization is sometimes needed in the internal algorithms (on hash collisions and sort-prefix collisions). The most efficient data types for that are actually Tuples. POJOs and other data types are a little less efficient for that, but we are trying to improve that.

Stephan

On Thu, Feb 19, 2015 at 10:23 AM, Kruse, Sebastian <[hidden email]> wrote:

Hi everyone,

 

I think that during one of the meetups, it was mentioned that Flink can in some cases operate on serialized data. Given I understood that correctly, which cases that would be, i.e, which data types and operators support such a feature?

 

Cheers,

Sebastian

 

 

---

Sebastian Kruse

Doktorand am Fachbereich Information Systems Group

Hasso-Plattner-Institut an der Universität Potsdam
Prof.-Dr.-Helmert-Str. 2-3, D-14482 Potsdam
Tel <a href="tel:%2B49%20331%205509%20240" value="[phone]" target="_blank">[phone]

Amtsgericht Potsdam, HRB 12184
Geschäftsführung: Prof. Dr. Christoph Meinel

 

 

[Plamen L. Simeonov]

Dear all,

can somebody of you help us with some information about ready to run examples for a fresh flink installation. We simply wish to know how it works when we feed it with some data (what comes out and how it performs).

many thanks!


___________________________________

Dr.-Ing. Plamen L. Simeonov
Department 1: Geodäsie und Fernerkundung
Sektion 1.5: Geoinformatik
Tel.: [phone]
Fax:  [phone]
email: [hidden email]
http://www.gfz-potsdam.de/
___________________________________

Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum - GFZ
Stiftung des öff. Rechts Land Brandenburg
Telegrafenberg A 20, 14473 Potsdam
**************************************************

[Plamen L. Simeonov]

Well, I got some bundled examples on the Flink page, but is there an elaborated test set with more details (e.g. Google keyword search simulation) around?

Thanks++

___________________________________

Dr.-Ing. Plamen L. Simeonov
Department 1: Geodäsie und Fernerkundung
Sektion 1.5: Geoinformatik
Tel.: [phone]
Fax:  [phone]
email: [hidden email]
http://www.gfz-potsdam.de/
___________________________________

Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum - GFZ
Stiftung des öff. Rechts Land Brandenburg
Telegrafenberg A 20, 14473 Potsdam
**************************************************

> On 19 Feb 2015, at 12:24, Plamen L. Simeonov <[hidden email]> wrote:
>
> Dear all,
>
> can somebody of you help us with some information about ready to run examples for a fresh flink installation. We simply wish to know how it works when we feed it with some data (what comes out and how it performs).
>
> many thanks!
>
>
> ___________________________________
>
> Dr.-Ing. Plamen L. Simeonov
> Department 1: Geodäsie und Fernerkundung
> Sektion 1.5: Geoinformatik
> Tel.: [phone]
> Fax:  [phone]
> email: [hidden email]
> http://www.gfz-potsdam.de/
> ___________________________________
>
> Helmholtz-Zentrum Potsdam
> Deutsches GeoForschungsZentrum - GFZ
> Stiftung des öff. Rechts Land Brandenburg
> Telegrafenberg A 20, 14473 Potsdam
> **************************************************
>
>

[rmetzger0]

Hi Plamen,

thank you for your interest in Flink!

It seems you've found already the flink examples. Please note that not all examples are contained in the "examples/" directory in of the binary release.

There are some more examples "hidden" in the source code:

Here are the Java examples: https://github.com/apache/flink/tree/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java

Scala: https://github.com/apache/flink/tree/master/flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala

Flink streaming (java): https://github.com/apache/flink/tree/master/flink-staging/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples

Flink Graph API (Gelly): https://github.com/apache/flink/tree/master/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example

Also, there is a repository that contains a bunch of testing jobs used by some of the Flink committers to test features: https://github.com/project-flink/flink-perf/tree/master/flink-jobs/src/main/java/com/github/projectflink

Please let me know if you have further questions regarding Flink.

Best regards,

Robert

On Thu, Feb 19, 2015 at 12:28 PM, Plamen L. Simeonov <[hidden email]> wrote:

Well, I got some bundled examples on the Flink page, but is there an elaborated test set with more details (e.g. Google keyword search simulation) around?

Thanks++

___________________________________

Dr.-Ing. Plamen L. Simeonov
Department 1: Geodäsie und Fernerkundung
Sektion 1.5: Geoinformatik
Tel.: <a href="tel:%2B49%20%280%29331%2F288-1587" value="[phone]">+49 (0)331/288-1587
Fax:  <a href="tel:%2B49%20%280%29331%2F288-1732" value="[phone]">+49 (0)331/288-1732
email: [hidden email]
http://www.gfz-potsdam.de/
___________________________________

Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum - GFZ
Stiftung des öff. Rechts Land Brandenburg
Telegrafenberg A 20, 14473 Potsdam
**************************************************

> On 19 Feb 2015, at 12:24, Plamen L. Simeonov <[hidden email]> wrote:
>
> Dear all,
>
> can somebody of you help us with some information about ready to run examples for a fresh flink installation. We simply wish to know how it works when we feed it with some data (what comes out and how it performs).
>
> many thanks!
>
>
> ___________________________________
>
> Dr.-Ing. Plamen L. Simeonov
> Department 1: Geodäsie und Fernerkundung
> Sektion 1.5: Geoinformatik
> Tel.: <a href="tel:%2B49%20%280%29331%2F288-1587" value="[phone]">+49 (0)331/288-1587
> Fax:  <a href="tel:%2B49%20%280%29331%2F288-1732" value="[phone]">+49 (0)331/288-1732
> email: [hidden email]
> http://www.gfz-potsdam.de/
> ___________________________________
>
> Helmholtz-Zentrum Potsdam
> Deutsches GeoForschungsZentrum - GFZ
> Stiftung des öff. Rechts Land Brandenburg
> Telegrafenberg A 20, 14473 Potsdam
> **************************************************
>
>