Data sink finishes before previous task finishes

[Chan, Regina]

Hi,

 

I’m still using Flink 1.2.1 on some production streams and I’m seeing this weird behavior where I’m seeing the flatMap stay in running while the data sink is in finished state. What’s causing this behavior?

 

 

 

 

Regina Chan

Goldman Sachs – Enterprise Platforms, Data Architecture

30 Hudson Street, 37th floor | Jersey City, NY 07302 (  [phone]

 

---

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

[Fabian Hueske-2]

Hi,

I don't know why this should happen. AFAIK, operators should transition to the finished state after their input operator finished.
Canceling in case of a failure is a different thing, but obviously not the case here.

This might be a bug.

@Till, do you know what might cause such behavior in Flink 1.2.x?

Best, Fabian

2018-05-22 21:03 GMT+02:00 Chan, Regina <[hidden email]>:

Hi,

 

I’m still using Flink 1.2.1 on some production streams and I’m seeing this weird behavior where I’m seeing the flatMap stay in running while the data sink is in finished state. What’s causing this behavior?

 

 

 

 

Regina Chan

Goldman Sachs – Enterprise Platforms, Data Architecture

30 Hudson Street, 37th floor | Jersey City, NY 07302 (  [phone]

 

---

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

[Till Rohrmann]

Hi Regina,

from the top of my hat, I cannot tell why an intermediate operator stays in state running while all others transitioned to a terminal state. Could you share the logs with us? They might be helpful in understanding the problem better.

Cheers,

Till

On Wed, May 23, 2018 at 12:44 PM, Fabian Hueske <[hidden email]> wrote:

Hi,

I don't know why this should happen. AFAIK, operators should transition to the finished state after their input operator finished.
Canceling in case of a failure is a different thing, but obviously not the case here.

This might be a bug.

@Till, do you know what might cause such behavior in Flink 1.2.x?

Best, Fabian

2018-05-22 21:03 GMT+02:00 Chan, Regina <[hidden email]>:

Hi,

 

I’m still using Flink 1.2.1 on some production streams and I’m seeing this weird behavior where I’m seeing the flatMap stay in running while the data sink is in finished state. What’s causing this behavior?

 

 

 

 

Regina Chan

Goldman Sachs – Enterprise Platforms, Data Architecture

30 Hudson Street, 37th floor | Jersey City, NY 07302 (  [phone]

 

---

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices