Comparing Flink vs Materialize

[Dan]

Has anyone compared Flink with Materialize?  A friend recommended me switch to Materialize.

In one of their blog posts, it says that Flink splits operators across CPUs (instead of splitting partitions across CPUs).  Is this true?  Is it configurable?

https://materialize.com/blog-rocksdb/

[Arvid Heise-3]

Hi Dan,

I have not touched Materialize yet, but the picture of them is too simplifying.

When you run Flink in parallel, then each source shard is assigned to one Flink source operator. Similarly, filter and map would run in parallel. Flink chains simple operators that have the same degree of parallelism by default, making them run on the same core. [1]

That means that their example looks exactly the same in Flink if you run Flink with parallelism 4. You need to exchange data only for aggregations with different keys (count in their example). The same is true for Kafka Streams btw.

So while there may be differences in how Materialize and Flink works, this example is not suitable to depict it.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#tasks-and-operator-chains

On Tue, Jan 5, 2021 at 2:03 AM Dan Hill <[hidden email]> wrote:

Has anyone compared Flink with Materialize?  A friend recommended me switch to Materialize.

In one of their blog posts, it says that Flink splits operators across CPUs (instead of splitting partitions across CPUs).  Is this true?  Is it configurable?

https://materialize.com/blog-rocksdb/

--