How to enable “upsert mode” for dynamic tables?

[Austin York]

I'm relatively new to Flink, so apologies in advance if this is a simple question.

I have seen several mentions of an "upsert mode" for dynamic tables based on a unique key in the Flink documentation and on the official Flink blog. However, I do not see any examples / documentation regarding how to enable this mode on a dynamic table.

Examples:

"When defining a dynamic table on a stream via update mode, we can specify a unique key attribute on the table. In that case, update and delete operations are performed with respect to the key attribute. The update mode is visualized in the following figure." (https://flink.apache.org/news/2017/04/04/dynamic-tables.html)

"A dynamic table that is converted into an upsert stream requires a (possibly composite) unique key." (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/streaming.html)

So my questions are:

How do I specify a unique key attribute on a dynamic table in Flink?

How do I place a dynamic table in update/upsert/"replace" mode, as opposed to append mode?

Posted in StackOverflow as well: https://stackoverflow.com/questions/48554999/apache-flink-how-to-enable-upsert-mode-for-dynamic-tables

Thanks in advance.

[Puneet Kinra-2]

As of now flink doesnt support this feature few days i came across the same requirement..

On Thu, Feb 1, 2018 at 9:55 AM, Austin York <[hidden email]> wrote:

I'm relatively new to Flink, so apologies in advance if this is a simple question.

I have seen several mentions of an "upsert mode" for dynamic tables based on a unique key in the Flink documentation and on the official Flink blog. However, I do not see any examples / documentation regarding how to enable this mode on a dynamic table.

Examples:

"When defining a dynamic table on a stream via update mode, we can specify a unique key attribute on the table. In that case, update and delete operations are performed with respect to the key attribute. The update mode is visualized in the following figure." (https://flink.apache.org/news/2017/04/04/dynamic-tables.html)

"A dynamic table that is converted into an upsert stream requires a (possibly composite) unique key." (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/streaming.html)

So my questions are:

How do I specify a unique key attribute on a dynamic table in Flink?

How do I place a dynamic table in update/upsert/"replace" mode, as opposed to append mode?

Posted in StackOverflow as well: https://stackoverflow.com/questions/48554999/apache-flink-how-to-enable-upsert-mode-for-dynamic-tables

Thanks in advance.

--

[Fabian Hueske-2]

Hi Austin,

thanks for your questions.

I posted an answer on Stack Overflow.

Let me know if you have further questions or comments.

Thanks, Fabian

2018-02-01 6:08 GMT+01:00 Puneet Kinra <[hidden email]>:

As of now flink doesnt support this feature few days i came across the same requirement..

On Thu, Feb 1, 2018 at 9:55 AM, Austin York <[hidden email]> wrote:

I'm relatively new to Flink, so apologies in advance if this is a simple question.

I have seen several mentions of an "upsert mode" for dynamic tables based on a unique key in the Flink documentation and on the official Flink blog. However, I do not see any examples / documentation regarding how to enable this mode on a dynamic table.

Examples:

"When defining a dynamic table on a stream via update mode, we can specify a unique key attribute on the table. In that case, update and delete operations are performed with respect to the key attribute. The update mode is visualized in the following figure." (https://flink.apache.org/news/2017/04/04/dynamic-tables.html)

"A dynamic table that is converted into an upsert stream requires a (possibly composite) unique key." (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/streaming.html)

So my questions are:

How do I specify a unique key attribute on a dynamic table in Flink?

How do I place a dynamic table in update/upsert/"replace" mode, as opposed to append mode?

Posted in StackOverflow as well: https://stackoverflow.com/questions/48554999/apache-flink-how-to-enable-upsert-mode-for-dynamic-tables

Thanks in advance.

--

[Austin York]

Thank you again for your detailed response, Fabian. I have this working now! 🎉

I have a follow up question that I'd love your input on, also posted on StackOverflow:

https://stackoverflow.com/questions/48591790/apache-flink-best-way-to-architect-many-to-one-join-on-dynamic-table

Again, I'm relatively new to Flink, so apologies in advance if this is a simple question.

Thanks,
Austin

On Thu, Feb 1, 2018 at 2:16 AM, Fabian Hueske <[hidden email]> wrote:

Hi Austin,

thanks for your questions.

I posted an answer on Stack Overflow.

Let me know if you have further questions or comments.

Thanks, Fabian

2018-02-01 6:08 GMT+01:00 Puneet Kinra <[hidden email]>:

As of now flink doesnt support this feature few days i came across the same requirement..

On Thu, Feb 1, 2018 at 9:55 AM, Austin York <[hidden email]> wrote:

I'm relatively new to Flink, so apologies in advance if this is a simple question.

I have seen several mentions of an "upsert mode" for dynamic tables based on a unique key in the Flink documentation and on the official Flink blog. However, I do not see any examples / documentation regarding how to enable this mode on a dynamic table.

Examples:

"When defining a dynamic table on a stream via update mode, we can specify a unique key attribute on the table. In that case, update and delete operations are performed with respect to the key attribute. The update mode is visualized in the following figure." (https://flink.apache.org/news/2017/04/04/dynamic-tables.html)

"A dynamic table that is converted into an upsert stream requires a (possibly composite) unique key." (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/streaming.html)

So my questions are:

How do I specify a unique key attribute on a dynamic table in Flink?

How do I place a dynamic table in update/upsert/"replace" mode, as opposed to append mode?

Posted in StackOverflow as well: https://stackoverflow.com/questions/48554999/apache-flink-how-to-enable-upsert-mode-for-dynamic-tables

Thanks in advance.

--