[DISCUSS] Drop connectors for 5.x and restart the flink es source connector

[jackylau]

Hi all:
     when i coding the es source connector here
https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
for the elasticsearch source connector. The doc is here
https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
,and i find  problem of SearchHits at

*currentScrollWindowHits = searchResponse.getHits().getHits()
*
And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
i use it in flink es connector base(the es dependency is 5). it will throw
this.
*Caused by: java.lang.IncompatibleClassChangeError: Found class
org.elasticsearch.search.SearchHits, but interface was expected.***

To fix it : we can do this ways

1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
Tuple2<String, String[]>, but it wll make the code weirdly

class ElasticsearchResponse
{
    String scroll;
    String[] result // convert ervery es connector
searchResponse.getHits().getHits() to this result
}


if user want to add some thing, it will need modify this
 

2) just support es 6,7 and upgrade flink-es-connector-base es dependency
version to 6 and drop flink-es-connector-5. And i found this discussion of
dropping es connector 2 and 5 here
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
.

 The es5 connector just support DataStream api currently .And Is it possible
to drop es5 connector and upgrade es-connector-base to es6?

I am looking forward all your response
Best !



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Jark Wu-3]

I'm fine with dropping support for es5. 

forward to dev@.

Best,

Jark

On Fri, 19 Jun 2020 at 15:46, jackylau <[hidden email]> wrote:

Hi all:
     when i coding the es source connector here
https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
for the elasticsearch source connector. The doc is here
https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
,and i find  problem of SearchHits at

*currentScrollWindowHits = searchResponse.getHits().getHits()
*
And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
i use it in flink es connector base(the es dependency is 5). it will throw
this.
*Caused by: java.lang.IncompatibleClassChangeError: Found class
org.elasticsearch.search.SearchHits, but interface was expected.***

To fix it : we can do this ways

1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
Tuple2<String, String[]>, but it wll make the code weirdly

class ElasticsearchResponse
{
    String scroll;
    String[] result // convert ervery es connector
searchResponse.getHits().getHits() to this result
}


if user want to add some thing, it will need modify this


2) just support es 6,7 and upgrade flink-es-connector-base es dependency
version to 6 and drop flink-es-connector-5. And i found this discussion of
dropping es connector 2 and 5 here
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
.

 The es5 connector just support DataStream api currently .And Is it possible
to drop es5 connector and upgrade es-connector-base to es6?

I am looking forward all your response
Best !



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[rmetzger0]

The discussion on dropping the ES5 connector was not conclusive, when we discussed it in February 2020. We wanted to revisit it for the 1.12 release.

From maven central, we have the following download numbers
ES2: 500 downloads

ES5: 10500 downloads (the es5_2.10:1.3.1 had 8000 downloads last month. I there's a CI system or something downloading all these)

ES6: 4200 downloads

ES7: 1800 downloads

For 1.10.0 we had the following numbers:
ES5: 500

ES6: 525

ES7: 840

Based on these numbers, I would advise against removing the ES5 connector for the 1.12 release.
 

On Fri, Jun 19, 2020 at 9:53 AM Jark Wu <[hidden email]> wrote:

I'm fine with dropping support for es5.

forward to dev@.

Best,
Jark



On Fri, 19 Jun 2020 at 15:46, jackylau <[hidden email]> wrote:

> Hi all:
>      when i coding the es source connector here
>
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> for the elasticsearch source connector. The doc is here
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> ,and
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and>
> i find  problem of SearchHits at
>
> *currentScrollWindowHits = searchResponse.getHits().getHits()
> *
> And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
> i use it in flink es connector base(the es dependency is 5). it will throw
> this.
> *Caused by: java.lang.IncompatibleClassChangeError: Found class
> org.elasticsearch.search.SearchHits, but interface was expected.***
>
> To fix it : we can do this ways
>
> 1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
> Tuple2<String, String[]>, but it wll make the code weirdly
>
> class ElasticsearchResponse
> {
>     String scroll;
>     String[] result // convert ervery es connector
> searchResponse.getHits().getHits() to this result
> }
>
>
> if user want to add some thing, it will need modify this
>
>
> 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> version to 6 and drop flink-es-connector-5. And i found this discussion of
> dropping es connector 2 and 5 here
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> .
>
>  The es5 connector just support DataStream api currently .And Is it
> possible
> to drop es5 connector and upgrade es-connector-base to es6?
>
> I am looking forward all your response
> Best !
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>