Hi All,
Does Flink 1.9 support create or replace views syntax in raw SQL? like spark streaming does? Thanks! |
I tried the following. bsTableEnv.sqlUpdate("CREATE VIEW my_view AS SELECT * FROM sample1 FULL OUTER JOIN sample2 on sample1.f0=sample2.f0");It looks like https://cwiki.apache.org/confluence/display/FLINK/FLIP-71+-+E2E+View+support+in+FLINK+SQL Views are not supported. Can I expect them to be supported in Flink 1.10? Currently, with Spark SQL when the query gets big I break it down into views and this is one of the most important features my application relies on. is there any workaround for this at the moment? Thanks! On Sat, Jan 18, 2020 at 6:24 PM kant kodali <[hidden email]> wrote:
|
Hi Kant, Sorry, 1.10 not support "CREATE VIEW" in raw SQL too. Workaround is: - Using TableEnvironment.createTemporaryView... - Or using "create view" and "drop view" in the sql-client. - Or using hive catalog, in 1.10, we support query catalog views. FLIP-71 will be finished in 1.11 soon. Best, Jingsong Lee On Sun, Jan 19, 2020 at 4:10 PM kant kodali <[hidden email]> wrote:
Best, Jingsong Lee |
Hi Jingsong, Thanks a lot, I think I can live with TableEnvironment.createTemporaryView in Flink 1.10 (which I am expecting to be released this month) but are these views persisted somewhere? for example across sessions? or say I stop my application and start again will it work as expected? Thanks! On Mon, Jan 20, 2020 at 1:12 AM Jingsong Li <[hidden email]> wrote:
|
Hi Kant,
The TableEnv#createTemporaryView and CREATE VIEW in SQL Cli both creates temporary views which is not persisted and will lost after session close. I think the persisted views will be supported in 1.11. Best, Jark
|
Hi Kant, If you want your view persisted, you must to dock a catalog like hive catalog, it stores views in the metastore with mysql. - In 1.10, you can store views in catalog through "Catalog.createTable", you can create a "CatalogViewImpl". This is an internal API, which is not easy to use. - In 1.11, we will introduce create view DDL for "TableEnv.sqlUpdate" and "TableEnv.createView". It will be easy to use. Best, Jingsong Lee On Tue, Jan 21, 2020 at 10:03 AM Jark Wu <[hidden email]> wrote:
Best, Jingsong Lee |
Hi Jingsong, Can I store it in Local Filesystem/HDFS? Thanks! On Mon, Jan 20, 2020 at 6:59 PM Jingsong Li <[hidden email]> wrote:
|
Hi Kant, if you want the store the catalog data in Local Filesystem/HDFS, you can implement a user defined catalog (just need to implement Catalog interface) Bests, Godfrey kant kodali <[hidden email]> 于2020年2月26日周三 下午12:28写道:
|
Hi, 2) If the views are not persisted what happens if the application crashes and restarted? will it create the view again and safely read the data from where it left off? Thanks! On Wed, Feb 26, 2020 at 6:47 AM godfrey he <[hidden email]> wrote:
|
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#hivecatalog Can I use the hive catalog to store view definitions in HDFS? I am assuming the metastore can be anything or does it have to be have MySQL? On Thu, Feb 27, 2020 at 4:46 AM kant kodali <[hidden email]> wrote:
|
HI kant, > 1) Where does the hive catalog persist view definitions? in mysql? or HDFS? hive catalog stores all metadata in derby or mysql [1] > 2) If the views are not persisted what happens if the application crashes and restarted? will it create the view again and safely read the data from where it left off? as Jark said, until to latest release version, the views are stored in memory. So if the application crashes or is restarted, you must create the view again. Bests, Godfrey kant kodali <[hidden email]> 于2020年2月27日周四 下午9:25写道:
|
Free forum by Nabble | Edit this page |