are blink changes merged into flink 1.9?

[kant kodali]

Hi All,

Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it?

Thanks!

[Benchao Li]

Hi kant,

Blink has merged most of it's functionality into 1.9-release, and we have two planners [1] in 1.9: 

• Blink Planner
• Old Planner (Legacy Planner)

You can try out blink planner by [2].

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#main-differences-between-the-two-planners

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#create-a-tableenvironment

kant kodali <[hidden email]> 于2020年1月12日周日 上午7:48写道：

Hi All,

Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it?

Thanks!

--

[kant kodali]

Hi,

Thanks for the response! The Documentation says Blink will treat batch as a special case of streaming. if so, can I stream from a database? or put it in another way. If I have two streams A & B where Stream A is writing to Database and Stream B reading from the database. will stream B keeps running forever picking up all the changes written to the database by Stream A? If not, is there any to achieve that sort of functionality?

The reason I may want to do something like this is I may want to do join on two tables that continuously get populated however I cannot specify the time constraint on the state which is why I would want to write to external database such that I can treat it as an infinite store and then do join as the data come in continuously. 

Thanks!

On Sat, Jan 11, 2020 at 6:42 PM Benchao Li <[hidden email]> wrote:

Hi kant,

Blink has merged most of it's functionality into 1.9-release, and we have two planners [1] in 1.9: 

• Blink Planner
• Old Planner (Legacy Planner)

You can try out blink planner by [2].

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#main-differences-between-the-two-planners

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#create-a-tableenvironment

kant kodali <[hidden email]> 于2020年1月12日周日 上午7:48写道：

Hi All,

Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it?

Thanks!

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]

[Benchao Li]

Hi kant,

In my understanding, "treat batch as a special case of streaming" just means runtime implementation. So currently, we cannot consume a database and it's changes as a stream. But You can put the change log into massage queue system like Kafka using a CDC system like Maxwell or Canal. Then you can consume it as a stream.

About the joining, maybe you can try Dimension Join[1] in blink.

Personally, I think reading from a database and it's continuous changes as a stream is a good idea, and I see some other users requesting this feature as well.

kant kodali <[hidden email]> 于2020年1月12日周日 下午8:07写道：

Hi,

Thanks for the response! The Documentation says Blink will treat batch as a special case of streaming. if so, can I stream from a database? or put it in another way. If I have two streams A & B where Stream A is writing to Database and Stream B reading from the database. will stream B keeps running forever picking up all the changes written to the database by Stream A? If not, is there any to achieve that sort of functionality?

The reason I may want to do something like this is I may want to do join on two tables that continuously get populated however I cannot specify the time constraint on the state which is why I would want to write to external database such that I can treat it as an infinite store and then do join as the data come in continuously. 

Thanks!

On Sat, Jan 11, 2020 at 6:42 PM Benchao Li <[hidden email]> wrote:

Hi kant,

Blink has merged most of it's functionality into 1.9-release, and we have two planners [1] in 1.9: 

• Blink Planner
• Old Planner (Legacy Planner)

You can try out blink planner by [2].

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#main-differences-between-the-two-planners

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#create-a-tableenvironment

kant kodali <[hidden email]> 于2020年1月12日周日 上午7:48写道：

Hi All,

Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it?

Thanks!

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]

--

[Benchao Li]

Sorry for the missing link.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/streaming/temporal_tables.html#temporal-table

Benchao Li <[hidden email]> 于2020年1月12日周日 下午9:06写道：

Hi kant,

In my understanding, "treat batch as a special case of streaming" just means runtime implementation. So currently, we cannot consume a database and it's changes as a stream. But You can put the change log into massage queue system like Kafka using a CDC system like Maxwell or Canal. Then you can consume it as a stream.

About the joining, maybe you can try Dimension Join[1] in blink.

Personally, I think reading from a database and it's continuous changes as a stream is a good idea, and I see some other users requesting this feature as well.

kant kodali <[hidden email]> 于2020年1月12日周日 下午8:07写道：

Hi,

Thanks for the response! The Documentation says Blink will treat batch as a special case of streaming. if so, can I stream from a database? or put it in another way. If I have two streams A & B where Stream A is writing to Database and Stream B reading from the database. will stream B keeps running forever picking up all the changes written to the database by Stream A? If not, is there any to achieve that sort of functionality?

The reason I may want to do something like this is I may want to do join on two tables that continuously get populated however I cannot specify the time constraint on the state which is why I would want to write to external database such that I can treat it as an infinite store and then do join as the data come in continuously. 

Thanks!

On Sat, Jan 11, 2020 at 6:42 PM Benchao Li <[hidden email]> wrote:

Hi kant,

Blink has merged most of it's functionality into 1.9-release, and we have two planners [1] in 1.9: 

• Blink Planner
• Old Planner (Legacy Planner)

You can try out blink planner by [2].

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#main-differences-between-the-two-planners

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#create-a-tableenvironment

kant kodali <[hidden email]> 于2020年1月12日周日 上午7:48写道：

Hi All,

Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it?

Thanks!

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]

--