Re: Async IO with SQL API

Posted by Till Rohrmann on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Async-IO-with-SQL-API-tp37500p37508.html

Hi Spurthi,

thanks for reaching out to the Flink community. Have you tried using the Blink planner where these features are available?

Pulling in Jark and Timo who worked on this feature and who might be able to explain to you the rationale behind making the LookupableTableSource and AsyncTableFunction a Blink planner feature only.

Cheers,

Till

On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <[hidden email]> wrote:

Hello folks,

    We are using flink 1.9 SQL API and we are NOT using blink planner. Our platform users express their flink jobs as SQL queries. We currently have a use case of asynchronously lookup data from third parties for every event we read from kafka stream and populate additional fields which we use later in our group bys.

We were trying to prototype the best way we can asynchronously enrich our events and two options we could think of:
    1. UDTF extending AsyncTableFunction and users can use these functions as part of SQL query. But it seems like AsyncTableFunction cannot be used as UDTF currently.
    2. LookupableTableSource to access data in Temporal table but its only supported in blink.

 Can you please suggest any other options I can try out?

Thanks,

-spurthi