REST API time out ( flink 1.5 ) on SP

[Vishal Santoshi]

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Till Rohrmann]

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Vishal Santoshi]

Aah sorry, while taking a save point without cancel, we hit this timeout  ( appears to be 10 seconds ).   The save point does succeed, in this case it takes roughly 13-15 seconds. Wanted to know which configuration to change to increase the time out on the REST call.  It does not seem to be  aka.client.timeout which is set to 120 s  which was the case in 1.4 and I could not  figure out from https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html which setting to work with.

Thanks much.

On Tue, Jul 10, 2018 at 3:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Till Rohrmann]

In order to configure the timeouts for the REST handlers, please use `web.timeout`. For the client timeout use `akka.client.timeout`.

Cheers,

Till

On Tue, Jul 10, 2018 at 10:54 AM Vishal Santoshi <[hidden email]> wrote:

Aah sorry, while taking a save point without cancel, we hit this timeout  ( appears to be 10 seconds ).   The save point does succeed, in this case it takes roughly 13-15 seconds. Wanted to know which configuration to change to increase the time out on the REST call.  It does not seem to be  aka.client.timeout which is set to 120 s  which was the case in 1.4 and I could not  figure out from https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html which setting to work with.

Thanks much.

On Tue, Jul 10, 2018 at 3:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Vishal Santoshi]

That should do.

Thanks much.

On Tue, Jul 10, 2018 at 7:52 AM, Till Rohrmann <[hidden email]> wrote:

In order to configure the timeouts for the REST handlers, please use `web.timeout`. For the client timeout use `akka.client.timeout`.

Cheers,

Till

On Tue, Jul 10, 2018 at 10:54 AM Vishal Santoshi <[hidden email]> wrote:

Aah sorry, while taking a save point without cancel, we hit this timeout  ( appears to be 10 seconds ).   The save point does succeed, in this case it takes roughly 13-15 seconds. Wanted to know which configuration to change to increase the time out on the REST call.  It does not seem to be  aka.client.timeout which is set to 120 s  which was the case in 1.4 and I could not  figure out from https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html which setting to work with.

Thanks much.

On Tue, Jul 10, 2018 at 3:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Vishal Santoshi]

As we on configurations let me take the liberty to ask this

Does 

akka.jvm-exit-on-fatal-error : true

have any relevance vis a vis quarantine ( it seems that we have our own gossip protocol )  and if not what other places is this used for it to be relevant ? 

On Tue, Jul 10, 2018 at 10:19 AM, Vishal Santoshi <[hidden email]> wrote:

That should do.

Thanks much.

On Tue, Jul 10, 2018 at 7:52 AM, Till Rohrmann <[hidden email]> wrote:

In order to configure the timeouts for the REST handlers, please use `web.timeout`. For the client timeout use `akka.client.timeout`.

Cheers,

Till

On Tue, Jul 10, 2018 at 10:54 AM Vishal Santoshi <[hidden email]> wrote:

Aah sorry, while taking a save point without cancel, we hit this timeout  ( appears to be 10 seconds ).   The save point does succeed, in this case it takes roughly 13-15 seconds. Wanted to know which configuration to change to increase the time out on the REST call.  It does not seem to be  aka.client.timeout which is set to 120 s  which was the case in 1.4 and I could not  figure out from https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html which setting to work with.

Thanks much.

On Tue, Jul 10, 2018 at 3:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..

[Till Rohrmann]

Hi, 

akka.jvm-exit-on-fatal-error says whether Akka will terminate the JVM if it encounters a fatal error (e.g. OOM). This has nothing todo with quarantining. The option is relevant for the legacy and new (FLIP-6) code base.

taskmanager.exit-on-fatal-akka-error is the configuration option which enables the quarantining monitor which kills the TaskManager JVM if it detects that the TaskManager has been quarantined or quarantined another system. This is only relevant for the legacy code base because with Flip-6 we no longer rely on Akka's death watch (main reason for quarantining).

Cheers,

Till

On Tue, Jul 10, 2018 at 4:47 PM Vishal Santoshi <[hidden email]> wrote:

As we on configurations let me take the liberty to ask this

Does 

akka.jvm-exit-on-fatal-error : true

have any relevance vis a vis quarantine ( it seems that we have our own gossip protocol )  and if not what other places is this used for it to be relevant ? 

On Tue, Jul 10, 2018 at 10:19 AM, Vishal Santoshi <[hidden email]> wrote:

That should do.

Thanks much.

On Tue, Jul 10, 2018 at 7:52 AM, Till Rohrmann <[hidden email]> wrote:

In order to configure the timeouts for the REST handlers, please use `web.timeout`. For the client timeout use `akka.client.timeout`.

Cheers,

Till

On Tue, Jul 10, 2018 at 10:54 AM Vishal Santoshi <[hidden email]> wrote:

Aah sorry, while taking a save point without cancel, we hit this timeout  ( appears to be 10 seconds ).   The save point does succeed, in this case it takes roughly 13-15 seconds. Wanted to know which configuration to change to increase the time out on the REST call.  It does not seem to be  aka.client.timeout which is set to 120 s  which was the case in 1.4 and I could not  figure out from https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html which setting to work with.

Thanks much.

On Tue, Jul 10, 2018 at 3:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

you need to give us a little bit more context in order to understand your question.

Cheers,

Till

On Mon, Jul 9, 2018 at 10:36 PM Vishal Santoshi <[hidden email]> wrote:

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager_10#-862909719]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".

  at .

which exact configuration is this..