Partial aggregation result sink

[李玥]

Hi,team
        I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?
For example, My SQL:
        select …
        from my_table
        GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!
LiYue

[chiggi_dev]

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue

[李玥]

Hi Chirag,

Thank for your reply!

I found a provided ContinuousEventTimeTrigger should be worked in my situation. 

Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? 

在 2018年3月12日，下午5:20，Chirag Dewan <[hidden email]> 写道：

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue

[Chesnay Schepler]

I don't think you can specify custom triggers when using purer SQL, but maybe Fabian or Timo know a SQL way of implementing your goal.

On 12.03.2018 13:16, 李玥 wrote:

Hi Chirag,

Thank for your reply!

I found a provided ContinuousEventTimeTrigger should be worked in my situation. 

Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? 

在 2018年3月12日，下午5:20，Chirag Dewan <[hidden email]> 写道：

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue

[Fabian Hueske]

Hi,

Chesnay is right.
SQL and Table API do not support early window results and no allowed lateness to update results with late arriving data.

If you need such features, you should use the DataStream API.

Best, Fabian

2018-03-13 12:10 GMT+01:00 Chesnay Schepler <[hidden email]>:

I don't think you can specify custom triggers when using purer SQL, but maybe Fabian or Timo know a SQL way of implementing your goal.

On 12.03.2018 13:16, 李玥 wrote:

Hi Chirag,

Thank for your reply!

I found a provided ContinuousEventTimeTrigger should be worked in my situation. 

Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? 

在 2018年3月12日，下午5:20，Chirag Dewan <[hidden email]> 写道：

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue

[李玥]

Hi ,

Is there any plan to adding these features to flink SQL ?

Thanks 

LiYue

tig.jd.com

在 2018年3月14日，上午7:48，Fabian Hueske <[hidden email]> 写道：

Hi,

Chesnay is right.
SQL and Table API do not support early window results and no allowed lateness to update results with late arriving data.

If you need such features, you should use the DataStream API.

Best, Fabian

2018-03-13 12:10 GMT+01:00 Chesnay Schepler <[hidden email]>:

I don't think you can specify custom triggers when using purer SQL, but maybe Fabian or Timo know a SQL way of implementing your goal.

On 12.03.2018 13:16, 李玥 wrote:

Hi Chirag,

Thank for your reply!

I found a provided ContinuousEventTimeTrigger should be worked in my situation. 

Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? 

在 2018年3月12日，下午5:20，Chirag Dewan <[hidden email]> 写道：

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue

[Fabian Hueske]

Hi,

Yes there are plans to support early results and control the result latency / result completeness trade off.

However, I cannot say when these features will be available.

Best, Fabian

2018-03-19 8:14 GMT+01:00 LiYue <[hidden email]>:

Hi ,

Is there any plan to adding these features to flink SQL ?

Thanks 

LiYue

tig.jd.com

在 2018年3月14日，上午7:48，Fabian Hueske <[hidden email]> 写道：

Hi,

Chesnay is right.
SQL and Table API do not support early window results and no allowed lateness to update results with late arriving data.

If you need such features, you should use the DataStream API.

Best, Fabian

2018-03-13 12:10 GMT+01:00 Chesnay Schepler <[hidden email]>:

I don't think you can specify custom triggers when using purer SQL, but maybe Fabian or Timo know a SQL way of implementing your goal.

On 12.03.2018 13:16, 李玥 wrote:

Hi Chirag,

Thank for your reply!

I found a provided ContinuousEventTimeTrigger should be worked in my situation. 

Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? 

在 2018年3月12日，下午5:20，Chirag Dewan <[hidden email]> 写道：

Hi LiYue,

This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows

Apache Flink 1.5-SNAPSHOT Documentation: Windows

So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:

  @Override

    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

      // schedule next timer

      ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);

      return TriggerResult.FIRE;

    }

I hope it helps.

Chirag

On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <[hidden email]> wrote:

Hi,team

    I’m working on a event-time based aggregation application with flink SQL.  Is there any way to keep sinking partial aggregation result BEFORE time window closed?

For example, My SQL:

    select …

    from my_table

    GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;

Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.

Thanks!

LiYue