I noticed that a simple SQL like 'insert into hive_parquet_table select … from some_kafka_table' will generates an additional operator called ’Sink: Unnamed’ with parallelism 1. I wonder if it’s by design? And what’s the functionality of this operator?
This is because for the sake of flexibility, the `StreamingFileCommitter` is implemented as a `StreamOperator` rather than a `SinkFunction`.
But `StreamTableSink` requires a `SinkFunction`, so we give a meaningless `DiscardingSink` to it. And this sink should be chained to upstream operator.
Best,
Jingsong
On Tue, Aug 4, 2020 at 5:03 PM Paul Lam <[hidden email]> wrote:
Hi,
I noticed that a simple SQL like 'insert into hive_parquet_table select … from some_kafka_table' will generates an additional operator called ’Sink: Unnamed’ with parallelism 1. I wonder if it’s by design? And what’s the functionality of this operator?