Extract column and table lineage from flink sql

[dawangli]

I want to build a lineage system for a real-time data warehouse，how can I extract table and column lineage from flink sql?

 

 

[Danny Chan-2]

Hi, dawangli ~

Usually people build the lineage of tables through a self-built platform, there was a DB to persist the relationship between the tables, for each job, you may need to analyze each SQL which are source tables and which are sink.

E.G. The INSERT target table is a sink and table after the scan or join is a source.

If you got the rel tree, you can get the info by a shuttle, if an AST instead, you can have a SqlVisitor.

Danny Chan <[hidden email]> 于2020年10月22日周四 下午5:24写道：

Hi, dawangli ~

Usually people build the lineage of tables through a self-built platform, there was a DB to persist the relationship between the tables, for each job, you may need to analyze each SQL which are source tables and which are sink.

E.G. The INSERT target table is a sink and table after the scan or join is a source.

If you got the rel tree, you can get the info by a shuttle, if an AST instead, you can have a SqlVisitor.

dawangli <[hidden email]> 于2020年10月20日周二 下午9:46写道：

I want to build a lineage system for a real-time data warehouse，how can I extract table and column lineage from flink sql?