(DEPRECATED) Apache Flink User Mailing List archive.

RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
kanagaraj.vengidasamy
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

kanagaraj.vengidasamy

Hi,

I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark  error frequently ...  and those task managers not processing messages after that error.
 What could be wrong in my configuration?  What I need to do to avoid this error.?

Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072


2016-12-26 14:26:06,548|  |WARN  io.netty.bootstrap.ServerBootstrap  - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
        at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
        at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
        at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)


Thanks
Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O 813.978.4372 | M 813.455.9757

    

Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Fabian Hueske-2
Hi Kanagaraj,

I would assume that the issue is caused by this configuration parameter:

taskmanager.memory.segment-size: 131072

I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.
There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.

Is there any particular reason why want to increase the network buffer size instead of keeping the default?
Did you find that the default parameter gives insufficient performance?

Best, Fabian

2016-12-26 20:52 GMT+01:00 <[hidden email]>:

Hi,

I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark  error frequently ...  and those task managers not processing messages after that error.
 What could be wrong in my configuration?  What I need to do to avoid this error.?

Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072


2016-12-26 14:26:06,548|  |WARN  io.netty.bootstrap.ServerBootstrap  - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
        at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
        at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
        at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)


Thanks
Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O 813.978.4372 | M 813.455.9757

    
kanagaraj.vengidasamy
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

kanagaraj.vengidasamy

Thanks Fabian,

 

                For default 65536 also got the same error . That is the reason increased to see it is helping or not.

 

Thanks


Verizon

Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O 813.978.4372 | M 813.455.9757

Facebook  Twitter  LinkedIn

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Tuesday, December 27, 2016 10:01 AM
To: [hidden email]
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

 

Hi Kanagaraj,

I would assume that the issue is caused by this configuration parameter:

taskmanager.memory.segment-size: 131072

I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.

There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.


Is there any particular reason why want to increase the network buffer size instead of keeping the default?

Did you find that the default parameter gives insufficient performance?

Best, Fabian

 

2016-12-26 20:52 GMT+01:00 <[hidden email]>:


Hi,

I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark  error frequently ...  and those task managers not processing messages after that error.
 What could be wrong in my configuration?  What I need to do to avoid this error.?

Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072


2016-12-26 14:26:06,548|  |WARN  io.netty.bootstrap.ServerBootstrap  - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
        at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
        at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
        at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)


Thanks
Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O 813.978.4372 | M 813.455.9757

    

 
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Fabian Hueske-2
Hi,

I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch.

The WARN log statement and the IllegalArgumentException are thrown by Netty.
Not sure what the implications are. My batch jobs finished successfully, so maybe the bad configuration options are just ignored.
Would be good to check this though.

Thanks,
Fabian

2016-12-27 16:10 GMT+01:00 <[hidden email]>:

Thanks Fabian,

 

                For default 65536 also got the same error . That is the reason increased to see it is helping or not.

 

Thanks


Verizon

Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O <a href="tel:(813)%20978-4372" value="+18139784372" target="_blank">813.978.4372 | M <a href="tel:(813)%20455-9757" value="+18134559757" target="_blank">813.455.9757

Facebook  Twitter  LinkedIn

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Tuesday, December 27, 2016 10:01 AM
To: [hidden email]
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

 

Hi Kanagaraj,

I would assume that the issue is caused by this configuration parameter:

taskmanager.memory.segment-size: 131072

I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.

There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.


Is there any particular reason why want to increase the network buffer size instead of keeping the default?

Did you find that the default parameter gives insufficient performance?

Best, Fabian

 

2016-12-26 20:52 GMT+01:00 <[hidden email]>:


Hi,

I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark  error frequently ...  and those task managers not processing messages after that error.
 What could be wrong in my configuration?  What I need to do to avoid this error.?

Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072


2016-12-26 14:26:06,548|  |WARN  io.netty.bootstrap.ServerBootstrap  - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
        at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
        at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
        at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)


Thanks
Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O <a href="tel:(813)%20978-4372" value="+18139784372" target="_blank">813.978.4372 | M <a href="tel:(813)%20455-9757" value="+18134559757" target="_blank">813.455.9757

    

 
kanagaraj.vengidasamy
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

kanagaraj.vengidasamy

Hi Fabian,

 

In my case, Since it is stream processing, after that error , that task manager  stuck and not taking any new messages. 

Can you  let me know how many task managers can I run  in 8 core x 32GB  machine.?  I am using Heap size as 4096 for each task manager.

 

Thanks


Verizon

Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O 813.978.4372 | M 813.455.9757

Facebook  Twitter  LinkedIn

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Tuesday, December 27, 2016 2:53 PM
To: [hidden email]
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

 

Hi,

I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch.

The WARN log statement and the IllegalArgumentException are thrown by Netty.

Not sure what the implications are. My batch jobs finished successfully, so maybe the bad configuration options are just ignored.

Would be good to check this though.

 

Thanks,

Fabian

 

2016-12-27 16:10 GMT+01:00 <[hidden email]>:

Thanks Fabian,

 

                For default 65536 also got the same error . That is the reason increased to see it is helping or not.

 

Thanks


Verizon

Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O <a href="tel:(813)%20978-4372" target="_blank">813.978.4372 | M <a href="tel:(813)%20455-9757" target="_blank"> 813.455.9757

Facebook  Twitter  LinkedIn

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Tuesday, December 27, 2016 10:01 AM
To: [hidden email]
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

 

Hi Kanagaraj,

I would assume that the issue is caused by this configuration parameter:

taskmanager.memory.segment-size: 131072

I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.

There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.


Is there any particular reason why want to increase the network buffer size instead of keeping the default?

Did you find that the default parameter gives insufficient performance?

Best, Fabian

 

2016-12-26 20:52 GMT+01:00 <[hidden email]>:


Hi,

I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark  error frequently ...  and those task managers not processing messages after that error.
 What could be wrong in my configuration?  What I need to do to avoid this error.?

Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072


2016-12-26 14:26:06,548|  |WARN  io.netty.bootstrap.ServerBootstrap  - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
        at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
        at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
        at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
        at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)


Thanks
Kanagaraj Vengidasamy
RTCI

7701 E Telecom PKWY
Temple Terrace, FL 33637

O <a href="tel:(813)%20978-4372" target="_blank">813.978.4372 | M <a href="tel:(813)%20455-9757" target="_blank"> 813.455.9757

    

 

 
Free forum by Nabble Edit this page