Some question about document

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Some question about document

Yuta Morisawa
Hi all

Now, I'm reading Flink document and I have some points to feel difficult
to get an idea.
I'd appreciate if you tell it me.

1,TypeInformation
  I understand TypeInformation is used for selecting relevant serializer
and comparator.
  But, the ducument doesn't specify if it has another way to be used.

  So, what I want to know is that what kinds of process gets benefit
from TypeInformation other than serializer and comparator.

2, Managed Memory
  The word "Managed memory" is appeared several time in the document
but I can't find any detail description.
  This is the only document I found
(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)

  If anyone has document that explains managed memory, please let me know.

3, Serializer
  What do the words in the document  "serializers we ship with Flink"
mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html


Regards,
Yuta

--

--------------------------------------------
  Business Vision :"Challenge for the future"
---------------------------------------------

   〒356-8502
 埼玉県ふじみ野市大原2丁目1番15号
 株式会社 KDDI総合研究所(KDDI Research, Inc.)
 コネクティッドカー1G
 森澤 雄太
 mail [hidden email]
 tel  070-3871-8883
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 この電子メールおよび添付書類は、名宛人のための
 特別な秘密情報を含んでおります。
 そのため、名宛人以外の方による利用は認められて
 おりません。
 名宛人以外の方による通信内容公表、複写、転用等
 は厳禁であり、違法となることがあります。
 万が一、何らかの誤りによりこの電子メールを名宛
 人以外の方が受信された場合は、お手数でも、直ち
 に発信人にお知らせ頂くと同時に、当メールを削除
 下さいますようお願い申し上げます。

Reply | Threaded
Open this post in threaded view
|

Re: Some question about document

Chesnay Schepler
1) TypeInformation are used to create serializers, comparators and to verify correctness of certain operations (like projections on tuple datasets).

2) see https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html

3) Flink comes with a number of serializers for varying types as outlined here.
For POJOs Flink has a custom serializer. For arbitrary objects we use kryo, and can use Avro as a fallback.

On 11.07.2018 09:24, Yuta Morisawa wrote:
Hi all

Now, I'm reading Flink document and I have some points to feel difficult to get an idea.
I'd appreciate if you tell it me.

1,TypeInformation
 I understand TypeInformation is used for selecting relevant serializer and comparator.
 But, the ducument doesn't specify if it has another way to be used.

 So, what I want to know is that what kinds of process gets benefit from TypeInformation other than serializer and comparator.

2, Managed Memory
 The word "Managed memory" is appeared several time in the document but I can't find any detail description.
 This is the only document I found (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)

 If anyone has document that explains managed memory, please let me know.

3, Serializer
 What do the words in the document  "serializers we ship with Flink" mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html


Regards,
Yuta


Reply | Threaded
Open this post in threaded view
|

Re: Some question about document

Yuta Morisawa
Thank you for your answer.

 > For POJOs Flink has a custom serializer. For arbitrary objects we use
 > kryo, and can use Avro as a fallback.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types

It may be the reverse.
Kryo is for fallback, right?


On 2018/07/11 19:00, Chesnay Schepler wrote:

> 1) TypeInformation are used to create serializers, comparators and to
> verify correctness of certain operations (like projections on tuple
> datasets).
>
> 2) see
> https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
>
> 3) Flink comes with a number of serializers for varying types as
> outlined here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>.
> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.
>
> On 11.07.2018 09:24, Yuta Morisawa wrote:
>> Hi all
>>
>> Now, I'm reading Flink document and I have some points to feel
>> difficult to get an idea.
>> I'd appreciate if you tell it me.
>>
>> 1,TypeInformation
>>  I understand TypeInformation is used for selecting relevant
>> serializer and comparator.
>>  But, the ducument doesn't specify if it has another way to be used.
>>
>>  So, what I want to know is that what kinds of process gets benefit
>> from TypeInformation other than serializer and comparator.
>>
>> 2, Managed Memory
>>  The word "Managed memory" is appeared several time in the document
>> but I can't find any detail description.
>>  This is the only document I found
>> (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)
>>
>>  If anyone has document that explains managed memory, please let me know.
>>
>> 3, Serializer
>>  What do the words in the document  "serializers we ship with Flink"
>> mean? I know Flink uses avro for POJOs, is it the same thing?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html 
>>
>>
>>
>> Regards,
>> Yuta
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Some question about document

vino yang
Hi Yuta,

It seems Chesnay is right. The "fallback" in flink's documentation is in terms of the types flink supported. But for all the other arbitrary types kryo is the first choice.

2018-07-12 9:55 GMT+08:00 Yuta Morisawa <[hidden email]>:
Thank you for your answer.

> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types

It may be the reverse.
Kryo is for fallback, right?


On 2018/07/11 19:00, Chesnay Schepler wrote:
1) TypeInformation are used to create serializers, comparators and to verify correctness of certain operations (like projections on tuple datasets).

2) see https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html

3) Flink comes with a number of serializers for varying types as outlined here <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>.
For POJOs Flink has a custom serializer. For arbitrary objects we use kryo, and can use Avro as a fallback.

On 11.07.2018 09:24, Yuta Morisawa wrote:
Hi all

Now, I'm reading Flink document and I have some points to feel difficult to get an idea.
I'd appreciate if you tell it me.

1,TypeInformation
 I understand TypeInformation is used for selecting relevant serializer and comparator.
 But, the ducument doesn't specify if it has another way to be used.

 So, what I want to know is that what kinds of process gets benefit from TypeInformation other than serializer and comparator.

2, Managed Memory
 The word "Managed memory" is appeared several time in the document but I can't find any detail description.
 This is the only document I found (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)

 If anyone has document that explains managed memory, please let me know.

3, Serializer
 What do the words in the document  "serializers we ship with Flink" mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html


Regards,
Yuta




Reply | Threaded
Open this post in threaded view
|

Re: Some question about document

Yuta Morisawa
Hi yang

Thank you for your comment.
I read document and got an idea.

Flink uses custom serializers on known types, and the fallback is kryo.
The other arbitary objects is serialized by kryo.



On 2018/07/12 12:14, vino yang wrote:

> Hi Yuta,
>
> It seems Chesnay is right. The "fallback" in flink's documentation is in
> terms of the types flink supported. But for all the other arbitrary
> types kryo is the first choice.
>
> 2018-07-12 9:55 GMT+08:00 Yuta Morisawa <[hidden email]
> <mailto:[hidden email]>>:
>
>     Thank you for your answer.
>
>     > For POJOs Flink has a custom serializer. For arbitrary objects we use
>     > kryo, and can use Avro as a fallback.
>     https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types
>     <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types>
>
>     It may be the reverse.
>     Kryo is for fallback, right?
>
>
>     On 2018/07/11 19:00, Chesnay Schepler wrote:
>
>         1) TypeInformation are used to create serializers, comparators
>         and to verify correctness of certain operations (like
>         projections on tuple datasets).
>
>         2) see
>         https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
>         <https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html>
>
>         3) Flink comes with a number of serializers for varying types as
>         outlined here
>         <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class
>         <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>>.
>         For POJOs Flink has a custom serializer. For arbitrary objects
>         we use kryo, and can use Avro as a fallback.
>
>         On 11.07.2018 09:24, Yuta Morisawa wrote:
>
>             Hi all
>
>             Now, I'm reading Flink document and I have some points to
>             feel difficult to get an idea.
>             I'd appreciate if you tell it me.
>
>             1,TypeInformation
>               I understand TypeInformation is used for selecting
>             relevant serializer and comparator.
>               But, the ducument doesn't specify if it has another way to
>             be used.
>
>               So, what I want to know is that what kinds of process gets
>             benefit from TypeInformation other than serializer and
>             comparator.
>
>             2, Managed Memory
>               The word "Managed memory" is appeared several time in the
>             document but I can't find any detail description.
>               This is the only document I found
>             (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi
>             <https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi>)
>
>               If anyone has document that explains managed memory,
>             please let me know.
>
>             3, Serializer
>               What do the words in the document  "serializers we ship
>             with Flink" mean? I know Flink uses avro for POJOs, is it
>             the same thing?
>             https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html
>             <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html>
>
>
>
>             Regards,
>             Yuta
>
>
>
>