DataStream to Table Api idioms

[swiesman]

Hi,

 

I am experimenting with rewriting some of my datastream projects with the table api and I had some questions on how to express certain idioms. I am using 1.4-SNAPSHOT.

 

1)       Can I express allowed lateness?

2)       Can I use a custom trigger? More specifically, I have a 24hr window but would like to receive partial results say every hour.

3)       Do window join time intervals have to be constant or can they depend on row attributes. I am running campaigns that have start and end dates and so I would like my join window to be that interval.

 

Thank you,

 

Seth Wiesman  

 

[Fabian Hueske-2]

Hi Seth,

I think the Table API is not there yet to address you use case.

1. Allowed lateness cannot be configured but it is on the list of features that we plan to add in the future.

2. Custom triggers are not supported. We are planning to add an option to support your use case (early firing and updates).

3. The window joins that will be release with 1.4 require constant boundaries (left.time > right.time - X and left.time < right.time + Y).
Variable join window boundaries have not been considered yet and would be quite tricky to implement. Would you mind opening a JIRA issue for this feature?

Best, Fabian

2017-11-06 14:56 GMT+01:00 Seth Wiesman <[hidden email]>:

Hi,

 

I am experimenting with rewriting some of my datastream projects with the table api and I had some questions on how to express certain idioms. I am using 1.4-SNAPSHOT.

 

1)       Can I express allowed lateness?

2)       Can I use a custom trigger? More specifically, I have a 24hr window but would like to receive partial results say every hour.

3)       Do window join time intervals have to be constant or can they depend on row attributes. I am running campaigns that have start and end dates and so I would like my join window to be that interval.

 

Thank you,

 

Seth Wiesman  

 

[swiesman]

Not a problem, thanks for the quick feedback.

 

https://issues.apache.org/jira/browse/FLINK-7999

 

Seth Wiesman

 

From: Fabian Hueske <[hidden email]>
Date: Monday, November 6, 2017 at 9:14 AM
To: Seth Wiesman <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: DataStream to Table Api idioms

 

Hi Seth,

 

I think the Table API is not there yet to address you use case.

 

1. Allowed lateness cannot be configured but it is on the list of features that we plan to add in the future.

2. Custom triggers are not supported. We are planning to add an option to support your use case (early firing and updates).

3. The window joins that will be release with 1.4 require constant boundaries (left.time > right.time - X and left.time < right.time + Y).
Variable join window boundaries have not been considered yet and would be quite tricky to implement. Would you mind opening a JIRA issue for this feature?

 

Best, Fabian

 

2017-11-06 14:56 GMT+01:00 Seth Wiesman <[hidden email]>:

Hi,

 

I am experimenting with rewriting some of my datastream projects with the table api and I had some questions on how to express certain idioms. I am using 1.4-SNAPSHOT.

 

1)       Can I express allowed lateness?

2)       Can I use a custom trigger? More specifically, I have a 24hr window but would like to receive partial results say every hour.

3)       Do window join time intervals have to be constant or can they depend on row attributes. I am running campaigns that have start and end dates and so I would like my join window to be that interval.

 

Thank you,

 

Seth Wiesman