Configuring Elasticsearch Timeouts

[Rex Fenley]

Hello,

For the past number of days we've been investigating continuous SocketTimeoutException like the following:

2021-01-12 20:53:42,105 DEBUG org.elasticsearch.client.RestClient                           - request [POST <our_host>:<port>/_bulk?timeout=1m] failed
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-01-12 20:53:42,106 DEBUG org.elasticsearch.client.RestClient                           - added [[host=<our_host>:<port>]] to blacklist

It appears that we're hitting a socket timeout and then the client adds our ES host to a blacklist which therefore fails any further request to ES and eventually fails our job.

Following from information we've gathered over the internet, we should be able to set the SocketTimeout to something higher than the HTTP timeout and it may fix the behavior we're seeing. However, we have not found a way to configure SocketTimeout from flink, what we want to set can be found here [1]. How do we set this from the Elasticsearch SQL connector?

[1] https://github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213

Thanks!

--

[Rex Fenley]

Correction, by HTTP timeout I mean BulkProcessor timeout.

On Tue, Jan 12, 2021 at 4:40 PM Rex Fenley <[hidden email]> wrote:

Hello,

For the past number of days we've been investigating continuous SocketTimeoutException like the following:

2021-01-12 20:53:42,105 DEBUG org.elasticsearch.client.RestClient                           - request [POST <our_host>:<port>/_bulk?timeout=1m] failed
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-01-12 20:53:42,106 DEBUG org.elasticsearch.client.RestClient                           - added [[host=<our_host>:<port>]] to blacklist

It appears that we're hitting a socket timeout and then the client adds our ES host to a blacklist which therefore fails any further request to ES and eventually fails our job.

Following from information we've gathered over the internet, we should be able to set the SocketTimeout to something higher than the HTTP timeout and it may fix the behavior we're seeing. However, we have not found a way to configure SocketTimeout from flink, what we want to set can be found here [1]. How do we set this from the Elasticsearch SQL connector?

[1] https://github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Rex Fenley]

For further clarity, we're on Elasticsearch 7.

On Tue, Jan 12, 2021 at 4:53 PM Rex Fenley <[hidden email]> wrote:

Correction, by HTTP timeout I mean BulkProcessor timeout.

On Tue, Jan 12, 2021 at 4:40 PM Rex Fenley <[hidden email]> wrote:

Hello,

For the past number of days we've been investigating continuous SocketTimeoutException like the following:

2021-01-12 20:53:42,105 DEBUG org.elasticsearch.client.RestClient                           - request [POST <our_host>:<port>/_bulk?timeout=1m] failed
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-01-12 20:53:42,106 DEBUG org.elasticsearch.client.RestClient                           - added [[host=<our_host>:<port>]] to blacklist

It appears that we're hitting a socket timeout and then the client adds our ES host to a blacklist which therefore fails any further request to ES and eventually fails our job.

Following from information we've gathered over the internet, we should be able to set the SocketTimeout to something higher than the HTTP timeout and it may fix the behavior we're seeing. However, we have not found a way to configure SocketTimeout from flink, what we want to set can be found here [1]. How do we set this from the Elasticsearch SQL connector?

[1] https://github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--