Async IO with SQL API

[Spurthi Chaganti]

Hello folks,

    We are using flink 1.9 SQL API and we are NOT using blink planner. Our platform users express their flink jobs as SQL queries. We currently have a use case of asynchronously lookup data from third parties for every event we read from kafka stream and populate additional fields which we use later in our group bys.

We were trying to prototype the best way we can asynchronously enrich our events and two options we could think of:
    1. UDTF extending AsyncTableFunction and users can use these functions as part of SQL query. But it seems like AsyncTableFunction cannot be used as UDTF currently.
    2. LookupableTableSource to access data in Temporal table but its only supported in blink.

 Can you please suggest any other options I can try out?

Thanks,

[Till Rohrmann]

Hi Spurthi,

thanks for reaching out to the Flink community. Have you tried using the Blink planner where these features are available?

Pulling in Jark and Timo who worked on this feature and who might be able to explain to you the rationale behind making the LookupableTableSource and AsyncTableFunction a Blink planner feature only.

Cheers,

Till

On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <[hidden email]> wrote:

Hello folks,

    We are using flink 1.9 SQL API and we are NOT using blink planner. Our platform users express their flink jobs as SQL queries. We currently have a use case of asynchronously lookup data from third parties for every event we read from kafka stream and populate additional fields which we use later in our group bys.

We were trying to prototype the best way we can asynchronously enrich our events and two options we could think of:
    1. UDTF extending AsyncTableFunction and users can use these functions as part of SQL query. But it seems like AsyncTableFunction cannot be used as UDTF currently.
    2. LookupableTableSource to access data in Temporal table but its only supported in blink.

 Can you please suggest any other options I can try out?

Thanks,

-spurthi

[Spurthi Chaganti]

Thank you Till for your response.

We haven't completely evaluated migration to blink planner as it was not prioritized, certainly a task we would look into if we arent able to get a solution using SQL api.

On Wed, Aug 19, 2020 at 3:23 AM Till Rohrmann <[hidden email]> wrote:

Hi Spurthi,

thanks for reaching out to the Flink community. Have you tried using the Blink planner where these features are available?

Pulling in Jark and Timo who worked on this feature and who might be able to explain to you the rationale behind making the LookupableTableSource and AsyncTableFunction a Blink planner feature only.

Cheers,

Till

On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <[hidden email]> wrote:

Hello folks,

    We are using flink 1.9 SQL API and we are NOT using blink planner. Our platform users express their flink jobs as SQL queries. We currently have a use case of asynchronously lookup data from third parties for every event we read from kafka stream and populate additional fields which we use later in our group bys.

We were trying to prototype the best way we can asynchronously enrich our events and two options we could think of:
    1. UDTF extending AsyncTableFunction and users can use these functions as part of SQL query. But it seems like AsyncTableFunction cannot be used as UDTF currently.
    2. LookupableTableSource to access data in Temporal table but its only supported in blink.

 Can you please suggest any other options I can try out?

Thanks,

-spurthi

--

[Jark Wu-3]

Hi,

Sorry for the late reply.

AFAIK, it's impossible to do Async IO on pure Table API / SQL in 1.9 old planner.

A doable way is convert the Table into DataStream and apply AsyncFunction on it. 

Best,

Jark

On Thu, 20 Aug 2020 at 00:35, Spurthi Chaganti <[hidden email]> wrote:

Thank you Till for your response.

We haven't completely evaluated migration to blink planner as it was not prioritized, certainly a task we would look into if we arent able to get a solution using SQL api.

On Wed, Aug 19, 2020 at 3:23 AM Till Rohrmann <[hidden email]> wrote:

Hi Spurthi,

thanks for reaching out to the Flink community. Have you tried using the Blink planner where these features are available?

Pulling in Jark and Timo who worked on this feature and who might be able to explain to you the rationale behind making the LookupableTableSource and AsyncTableFunction a Blink planner feature only.

Cheers,

Till

On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <[hidden email]> wrote:

Hello folks,

    We are using flink 1.9 SQL API and we are NOT using blink planner. Our platform users express their flink jobs as SQL queries. We currently have a use case of asynchronously lookup data from third parties for every event we read from kafka stream and populate additional fields which we use later in our group bys.

We were trying to prototype the best way we can asynchronously enrich our events and two options we could think of:
    1. UDTF extending AsyncTableFunction and users can use these functions as part of SQL query. But it seems like AsyncTableFunction cannot be used as UDTF currently.
    2. LookupableTableSource to access data in Temporal table but its only supported in blink.

 Can you please suggest any other options I can try out?

Thanks,

-spurthi

--

-spurthi