Re: How do I ensure binary comparisons are being used?
Posted by
Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-do-I-ensure-binary-comparisons-are-being-used-tp10806p10837.html
Hi Lawrence,
comparison of binary data are mainly used by the DataSet API when sorting large data sets or building and probing hash tables.
The DataStream API mainly benefits from Flink's custom and efficient serialization when sending data over the wire or taking checkpoints.
There are also plans to implement a state backend based on the serialization stack which leverages Flink's managed memory instead of holding object on the heap (the RocksDB state backend is the current solution to avoid this).
From what I know, the DataStream API does not perform compare on serialized data.
Best, Fabian