[DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Dawid Wysakowicz-2]

Hi all,

As described in this https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced in 5.2.

Both versions are already long eol: https://www.elastic.co/support/eol

I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

[rmetzger0]

Thanks for starting this discussion!

+1 to drop both

On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi all,

As described in this https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced in 5.2.

Both versions are already long eol: https://www.elastic.co/support/eol

I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

[Itamar Syn-Hershko]

+1 from dropping old versions because of jar hells etc. However, in the wild there are still a lot of 2.x clusters and definitely 5.x clusters that are having a hard time upgrading. We know because we assist those on a daily basis.

It is very easy to create an HTTP based connector that works with all ES versions, though. As Elasticsearch consultants and experts we have done that many times before. For example see this simplified client that has zero dependencies and can be easily brought in to Flink to use as a sink for all ES versions: https://github.com/BigDataBoutique/log4j2-elasticsearch-http/blob/master/src/main/java/com/bigdataboutique/logging/log4j2/ElasticsearchHttpClient.java

Will be happy to assist in such effort

On Mon, Feb 10, 2020 at 3:45 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi all,

As described in this https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced in 5.2.

Both versions are already long eol: https://www.elastic.co/support/eol

I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

--

[Flavio Pompermaier]

+1 for dropping all Elasticsearch connectors < 6.x

On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi all,

As described in this https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced in 5.2.

Both versions are already long eol: https://www.elastic.co/support/eol

I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

[Benchao Li]

+1 for dropping 2.x - 5.x.  

FYI currently only 6.x and 7.x ES Connectors are supported by table api.

Flavio Pompermaier <[hidden email]> 于2020年2月10日周一 下午10:03写道：

+1 for dropping all Elasticsearch connectors < 6.x

On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]>
wrote:

> Hi all,
>
> As described in this https://issues.apache.org/jira/browse/FLINK-11720
> ticket our elasticsearch 5.x connector does not work out of the box on
> some systems and requires a version bump. This also happens for our e2e.
> We cannot bump the version in es 5.x connector, because 5.x connector
> shares a common class with 2.x that uses an API that was replaced in 5.2.
>
> Both versions are already long eol: https://www.elastic.co/support/eol
>
> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> both of them, I would strongly suggest dropping at least 2.x connector
> and update the 5.x line to a working es client module.
>
> What do you think? Should we drop both versions? Drop only the 2.x
> connector? Or keep them both?
>
> Best,
>
> Dawid
>
>

--