a lot of connections in state "CLOSE_WAIT"

[XiangWei Huang]

There were a lot of connections which state  is “CLOSE_WAIT”  appeared when a few days later after flink jobManager is started.

The process name of pid 20180 is 'flink jobManager'.

The stack trace is blow:

[Nico Kruber]

Hi XiangWei,
this could be a resource leak, i.e. a socket not getting closed, but I was
unable to reproduce that behaviour. Maybe Chesnay (cc'd) has an idea on how/
where this may happen.

Can you tell us a bit more on what you where doing / how the webinterface was
used? Is there a way to reproduce the behaviour?



Nico


PS: please don't post text output as images - this makes reading/finding the
email and following the thread much more complicated. Please copy the text
instead.

On Thursday, 29 June 2017 06:36:47 CEST XiangWei Huang wrote:
> There were a lot of connections which state  is “CLOSE_WAIT”  appeared when
> a few days later after flink jobManager is started.
>
> The process name of pid 20180 is 'flink jobManager'.
> The stack trace is blow:

[Chesnay Schepler]

Hello,

is this list constantly growing? If you reload the WebUI do they pile up
again?

My guess would be the watermarks display is overloading the metrics handler.
If i remember correctly the WebUI keeps fetching the watermark metrics
regardless
of what page you're looking at.

Things to investigate would be whether it continues to query for
watermarks for finished
jobs, and whether requests are batched across tasks & jobs.

Regards,
Chesnay

On 30.06.2017 18:05, Nico Kruber wrote:

> Hi XiangWei,
> this could be a resource leak, i.e. a socket not getting closed, but I was
> unable to reproduce that behaviour. Maybe Chesnay (cc'd) has an idea on how/
> where this may happen.
>
> Can you tell us a bit more on what you where doing / how the webinterface was
> used? Is there a way to reproduce the behaviour?
>
>
>
> Nico
>
>
> PS: please don't post text output as images - this makes reading/finding the
> email and following the thread much more complicated. Please copy the text
> instead.
>
> On Thursday, 29 June 2017 06:36:47 CEST XiangWei Huang wrote:
>> There were a lot of connections which state  is “CLOSE_WAIT”  appeared when
>> a few days later after flink jobManager is started.
>>
>> The process name of pid 20180 is 'flink jobManager'.
>> The stack trace is blow:

[XiangWei Huang]

hi,

Sorry for replying so late.
I have met this issue again and the list is constantly keep growing even if i close the page ,until the website is been unavailable.

This issue appeared each time i add  metrics for a job from web ui.

by the way ,the version of Flink is 1.3.1



Regards,
XiangWei

[Chesnay Schepler]

Hello,

Could you tell us which browser you are using, including the version?
(and maybe try out if the issue persists with a different one)

Regards,
Chesnay

On 25.07.2017 05:20, XiangWei Huang wrote:

> hi,
>
> Sorry for replying so late.
> I have met this issue again and the list is constantly keep growing even if
> i close the page ,until the website is been unavailable.
>
> This issue appeared each time i add  metrics for a job from web ui.
>
> by the way ,the version of Flink is 1.3.1
>
>
>
> Regards,
> XiangWei
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>

[XiangWei Huang]

hi,

The browser i am using is Google Chrome  with version 59.0.3071.115 and the issue persists when i tried Firefox.

Regards,
XiangWei

> 在 2017年7月25日，17:48，Chesnay Schepler <[hidden email]> 写道：
>
> Hello,
>
> Could you tell us which browser you are using, including the version?
> (and maybe try out if the issue persists with a different one)
>
> Regards,
> Chesnay
>
> On 25.07.2017 05:20, XiangWei Huang wrote:
>> hi,
>>
>> Sorry for replying so late.
>> I have met this issue again and the list is constantly keep growing even if
>> i close the page ,until the website is been unavailable.
>>
>> This issue appeared each time i add  metrics for a job from web ui.
>>
>> by the way ,the version of Flink is 1.3.1
>>
>>
>>
>> Regards,
>> XiangWei
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
>> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>>
>

[Chesnay Schepler]

So this only happens when you select a metric? Without a selected metric everything works fine?

Are the metrics you selected shown correctly?

Did you modify the "jobmanager.web.refresh-interval" setting? (maybe check the flink-conf-yaml for the current setting)

On 26.07.2017 04:57, XiangWei Huang wrote:

hi,

The browser i am using is Google Chrome  with version 59.0.3071.115 and the issue persists when i tried Firefox.

Regards,
XiangWei

在 2017年7月25日，17:48，Chesnay Schepler [hidden email] 写道：

Hello,

Could you tell us which browser you are using, including the version?
(and maybe try out if the issue persists with a different one)

Regards,
Chesnay

On 25.07.2017 05:20, XiangWei Huang wrote:

hi,

Sorry for replying so late.
I have met this issue again and the list is constantly keep growing even if
i close the page ,until the website is been unavailable.

This issue appeared each time i add  metrics for a job from web ui.

by the way ,the version of Flink is 1.3.1



Regards,
XiangWei




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[XiangWei Huang]

1. yes and yes.

2. Yes,it was shown correctly.

3.I wasn’t modify this setting.

在 2017年7月26日，18:06，Chesnay Schepler [via Apache Flink User Mailing List archive.] <[hidden email]> 写道：

So this only happens when you select a metric? Without a selected metric everything works fine?

Are the metrics you selected shown correctly?

Did you modify the "jobmanager.web.refresh-interval" setting? (maybe check the flink-conf-yaml for the current setting)

On 26.07.2017 04:57, XiangWei Huang wrote:

hi,

The browser i am using is Google Chrome  with version 59.0.3071.115 and the issue persists when i tried Firefox.

Regards,
XiangWei

在 2017年7月25日，17:48，Chesnay Schepler <a href="x-msg://6/user/SendEmail.jtp?type=node&amp;node=14463&amp;i=0" target="_top" rel="nofollow" link="external" class="">[hidden email] 写道：

Hello,

Could you tell us which browser you are using, including the version?
(and maybe try out if the issue persists with a different one)

Regards,
Chesnay

On 25.07.2017 05:20, XiangWei Huang wrote:

hi,

Sorry for replying so late.
I have met this issue again and the list is constantly keep growing even if
i close the page ,until the website is been unavailable.

This issue appeared each time i add  metrics for a job from web ui.

by the way ,the version of Flink is 1.3.1



Regards,
XiangWei




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14463.html

To unsubscribe from a lot of connections in state "CLOSE_WAIT", click here.
NAML

[Chesnay Schepler]

FLINK-7368 may be the reason for this behavior.

On 31.07.2017 03:54, XiangWei Huang wrote:

1. yes and yes.

2. Yes,it was shown correctly.

3.I wasn’t modify this setting.

在 2017年7月26日，18:06，Chesnay Schepler [via Apache Flink User Mailing List archive.] <[hidden email]> 写道：

So this only happens when you select a metric? Without a selected metric everything works fine?

Are the metrics you selected shown correctly?

Did you modify the "jobmanager.web.refresh-interval" setting? (maybe check the flink-conf-yaml for the current setting)

On 26.07.2017 04:57, XiangWei Huang wrote:

hi,

The browser i am using is Google Chrome  with version 59.0.3071.115 and the issue persists when i tried Firefox.

Regards,
XiangWei

在 2017年7月25日，17:48，Chesnay Schepler <a href="x-msg://6/user/SendEmail.jtp?type=node&amp;node=14463&amp;i=0" target="_top" rel="nofollow" link="external" class="">[hidden email] 写道：

Hello,

Could you tell us which browser you are using, including the version?
(and maybe try out if the issue persists with a different one)

Regards,
Chesnay

On 25.07.2017 05:20, XiangWei Huang wrote:

hi,

Sorry for replying so late.
I have met this issue again and the list is constantly keep growing even if
i close the page ,until the website is been unavailable.

This issue appeared each time i add  metrics for a job from web ui.

by the way ,the version of Flink is 1.3.1



Regards,
XiangWei




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14463.html

To unsubscribe from a lot of connections in state "CLOSE_WAIT", click here.
NAML

---

View this message in context: Re: a lot of connections in state "CLOSE_WAIT"
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[XiangWei Huang]

It seems so,thanks for your reply Chesnay.

在 2017年8月8日，22:23，Chesnay Schepler [via Apache Flink User Mailing List archive.] <[hidden email]> 写道：

FLINK-7368 may be the reason for this behavior.

On 31.07.2017 03:54, XiangWei Huang wrote:

1. yes and yes.

2. Yes,it was shown correctly.

3.I wasn’t modify this setting.

在 2017年7月26日，18:06，Chesnay Schepler [via Apache Flink User Mailing List archive.] <<a href="x-msg://16/user/SendEmail.jtp?type=node&amp;node=14539&amp;i=0" target="_top" rel="nofollow" link="external" moz-do-not-send="true" class="">[hidden email]> 写道：

So this only happens when you select a metric? Without a selected metric everything works fine?

Are the metrics you selected shown correctly?

Did you modify the "jobmanager.web.refresh-interval" setting? (maybe check the flink-conf-yaml for the current setting)

On 26.07.2017 04:57, XiangWei Huang wrote:

hi,

The browser i am using is Google Chrome  with version 59.0.3071.115 and the issue persists when i tried Firefox.

Regards,
XiangWei

在 2017年7月25日，17:48，Chesnay Schepler <a href="<a href="x-msg://6/user/SendEmail.jtp?type=node&amp;amp;node=14463&amp;amp;i=0" class="">x-msg://6/user/SendEmail.jtp?type=node&amp;node=14463&amp;i=0" target="_top" rel="nofollow" link="external" class="">[hidden email] 写道：

Hello,

Could you tell us which browser you are using, including the version?
(and maybe try out if the issue persists with a different one)

Regards,
Chesnay

On 25.07.2017 05:20, XiangWei Huang wrote:

hi,

Sorry for replying so late.
I have met this issue again and the list is constantly keep growing even if
i close the page ,until the website is been unavailable.

This issue appeared each time i add  metrics for a job from web ui.

by the way ,the version of Flink is 1.3.1



Regards,
XiangWei




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14422.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14463.html

To unsubscribe from a lot of connections in state "CLOSE_WAIT", click here.
NAML

---

View this message in context: Re: a lot of connections in state "CLOSE_WAIT"
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/a-lot-of-connections-in-state-CLOSE-WAIT-tp14046p14740.html

To unsubscribe from a lot of connections in state "CLOSE_WAIT", click here.
NAML