Pulsar as a state backend

[Michael Colson]

Hello,

I recently browse this post : 

https://flink.apache.org/2019/05/03/pulsar-flink.html

and mainly :

Finally, an alternative way of integrating the technologies could include using Pulsar as a state backend with Flink. Since Pulsar has a layered architecture (Streams and Segmented Streams, powered by Apache Bookkeeper), it becomes natural to use Pulsar as a storage layer and store Flink state.

Are there any advanced specifications or efforts in the current roadmap for this integration ? 

Thanks,

---

You received this electronic message as part of a business or employment relationship with one or several S4M entities. Its content is strictly confidential and is covered by the obligation of confidentiality and business secrecy. Any dissemination, copying, printing distribution, retention or use of the message’s content or any attachments that could be detrimental to S4M is forbidden, even if it was forwarded by mailing lists. 

If you are not the intended recipient, please notify the sender of the error without delay and delete permanently this email and any files from your system and destroy any printed copies. 

[Markos Sfikas]

Hi Michael, 

Thanks for the question regarding the Pulsar Connector. 

The team is currently focusing on further developing the Flink-Pulsar connector with the following items: 

• Addition of a columnar off-loader to improve the performance for batch queries.The columnar segment off-loader is currently in progress and expected to be out sometime in May.
• Work on state integration is also in progress and expected sometime in May. 
• Using Pulsar as a Flink state backend is something that the team will also be working on once the off-loader (mentioned above) is complete so this should be expected sometime in June. 

All such development efforts will be part of the Pulsar Connector, with the intention to be contributed back to the Flink community. 

You can follow the progress across all development efforts in FLIP-72[1]

I hope this helps. 

Best 

Markos 

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector

On Mon, 6 Apr 2020 at 17:24, Michael Colson <[hidden email]> wrote:

Hello,

I recently browse this post : 

https://flink.apache.org/2019/05/03/pulsar-flink.html

and mainly :

Finally, an alternative way of integrating the technologies could include using Pulsar as a state backend with Flink. Since Pulsar has a layered architecture (Streams and Segmented Streams, powered by Apache Bookkeeper), it becomes natural to use Pulsar as a storage layer and store Flink state.

Are there any advanced specifications or efforts in the current roadmap for this integration ? 

Thanks,

---

You received this electronic message as part of a business or employment relationship with one or several S4M entities. Its content is strictly confidential and is covered by the obligation of confidentiality and business secrecy. Any dissemination, copying, printing distribution, retention or use of the message’s content or any attachments that could be detrimental to S4M is forbidden, even if it was forwarded by mailing lists. 

If you are not the intended recipient, please notify the sender of the error without delay and delete permanently this email and any files from your system and destroy any printed copies. 

[Michael Colson]

Hello Markos,

Le mar. 7 avr. 2020 à 03:50, Markos Sfikas <[hidden email]> a écrit :

Hi Michael, 

Thanks for the question regarding the Pulsar Connector. 

The team is currently focusing on further developing the Flink-Pulsar connector with the following items: 

• Addition of a columnar off-loader to improve the performance for batch queries.The columnar segment off-loader is currently in progress and expected to be out sometime in May.
• Work on state integration is also in progress and expected sometime in May. 
• Using Pulsar as a Flink state backend is something that the team will also be working on once the off-loader (mentioned above) is complete so this should be expected sometime in June. 

All such development efforts will be part of the Pulsar Connector, with the intention to be contributed back to the Flink community. 

You can follow the progress across all development efforts in FLIP-72[1]

I hope this helps. 

Best 

Markos 

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector

On Mon, 6 Apr 2020 at 17:24, Michael Colson <[hidden email]> wrote:

Hello,

I recently browse this post : 

https://flink.apache.org/2019/05/03/pulsar-flink.html

and mainly :

Finally, an alternative way of integrating the technologies could include using Pulsar as a state backend with Flink. Since Pulsar has a layered architecture (Streams and Segmented Streams, powered by Apache Bookkeeper), it becomes natural to use Pulsar as a storage layer and store Flink state.

Are there any advanced specifications or efforts in the current roadmap for this integration ? 

Thanks,

---

You received this electronic message as part of a business or employment relationship with one or several S4M entities. Its content is strictly confidential and is covered by the obligation of confidentiality and business secrecy. Any dissemination, copying, printing distribution, retention or use of the message’s content or any attachments that could be detrimental to S4M is forbidden, even if it was forwarded by mailing lists. 

If you are not the intended recipient, please notify the sender of the error without delay and delete permanently this email and any files from your system and destroy any printed copies. 

---

You received this electronic message as part of a business or employment relationship with one or several S4M entities. Its content is strictly confidential and is covered by the obligation of confidentiality and business secrecy. Any dissemination, copying, printing distribution, retention or use of the message’s content or any attachments that could be detrimental to S4M is forbidden, even if it was forwarded by mailing lists. 

If you are not the intended recipient, please notify the sender of the error without delay and delete permanently this email and any files from your system and destroy any printed copies.