Job-level close()?

[Andrew Roberts]

Hello,

I’m writing a Flink operator that connects to a database, and running it in parallel results in issues due to the singleton nature of the connection pool in the library I’m working with. The operator needs to close the connection pool when it’s done, but only when ALL parallel instances are done. If one subtask finishes first, then it closes the pool out from under the subtasks that are still working. Currently, I’m using a reference-counting hack that’s pretty brittle and unsatisfying. Are there any plans to add a user-definable job-level cleanup interface?

Thanks

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*

[Fabian Hueske-2]

Hi Andrew,

I'm not aware of such a plan.

Another way to address such issues is to run multiple TaskManagers with a single slot. In that case, only one subtask is executed per TM processes.

Best, Fabian

2017-12-15 22:23 GMT+01:00 Andrew Roberts <[hidden email]>:

Hello,

I’m writing a Flink operator that connects to a database, and running it in parallel results in issues due to the singleton nature of the connection pool in the library I’m working with. The operator needs to close the connection pool when it’s done, but only when ALL parallel instances are done. If one subtask finishes first, then it closes the pool out from under the subtasks that are still working. Currently, I’m using a reference-counting hack that’s pretty brittle and unsatisfying. Are there any plans to add a user-definable job-level cleanup interface?

Thanks

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*

[Eron Wright]

Since the connection pool is per-TM, and a given job may span numerous TMs, I think you're imagining a per-TM cleanup.

Your idea of using reference counting seems like the best option.  Maintain a static count of open instances.  Use a synchronization block to manage the count and to conditionally open and close the connection pool.

On Mon, Dec 18, 2017 at 1:58 AM, Fabian Hueske <[hidden email]> wrote:

Hi Andrew,

I'm not aware of such a plan.

Another way to address such issues is to run multiple TaskManagers with a single slot. In that case, only one subtask is executed per TM processes.

Best, Fabian

2017-12-15 22:23 GMT+01:00 Andrew Roberts <[hidden email]>:

Hello,

I’m writing a Flink operator that connects to a database, and running it in parallel results in issues due to the singleton nature of the connection pool in the library I’m working with. The operator needs to close the connection pool when it’s done, but only when ALL parallel instances are done. If one subtask finishes first, then it closes the pool out from under the subtasks that are still working. Currently, I’m using a reference-counting hack that’s pretty brittle and unsatisfying. Are there any plans to add a user-definable job-level cleanup interface?

Thanks

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*