Secure TLS/SSL ElasticSearch connector for current and future connector

[Fritz Budiyanto]

Hi All,

Anyone know if Flink has TLS/SSL support for the current ES connector ?
If yes, any sample configuration/code ?
If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ?

Thanks,
Fritz

[Christophe Jolif]

Hi Fritz,

I think the High Level Rest Client implementation in this PR: https://github.com/apache/flink/pull/5374 should work. If you don't get the certificate properly available in your Java certs, you might want to redefine the createClient method to do something along those lines to get the context aware of it:

https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_encrypted_communication.html

We might want to amend the code to make that even easier (and also manage basic auth: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html)

That said I'm unsure how the community wants to pursue on this next generation ES connector, because despite the obvious need from the number of requests, it does not seem there is a lot of traction to get something actually merged.

My current thinking would be to build a brand new Java REST High Level client-only based client, possibly breaking some compatibility with old APIs (that said my PR above it trying to keep the compatibility to the price of a few cast). This would leave undone the 5.2-6.0 support. And start back working on 6.1+. But at least there would be something "correct" for the future.

--

Christophe

On Mon, Mar 26, 2018 at 11:38 PM, Fritz Budiyanto <[hidden email]> wrote:

Hi All,

Anyone know if Flink has TLS/SSL support for the current ES connector ?
If yes, any sample configuration/code ?
If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ?

Thanks,
Fritz

[Fritz Budiyanto]

Hi Christophe,

Thanks so much for the pointers. That helps.

Looking at the latest update on https://issues.apache.org/jira/browse/FLINK-8101, there was an issue related to HLR retry handling. If I read this correctly, there is a bug in ES/HLR and some tests were failed because of that and hence this PR cant be merged. 

Also have you tried HLR based connector at scale, is it stable?

--

Fritz

On Mar 26, 2018, at 3:18 PM, Christophe Jolif <[hidden email]> wrote:

Hi Fritz,

I think the High Level Rest Client implementation in this PR: https://github.com/apache/flink/pull/5374 should work. If you don't get the certificate properly available in your Java certs, you might want to redefine the createClient method to do something along those lines to get the context aware of it:

https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_encrypted_communication.html

We might want to amend the code to make that even easier (and also manage basic auth: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html)

That said I'm unsure how the community wants to pursue on this next generation ES connector, because despite the obvious need from the number of requests, it does not seem there is a lot of traction to get something actually merged.

My current thinking would be to build a brand new Java REST High Level client-only based client, possibly breaking some compatibility with old APIs (that said my PR above it trying to keep the compatibility to the price of a few cast). This would leave undone the 5.2-6.0 support. And start back working on 6.1+. But at least there would be something "correct" for the future.

--

Christophe

On Mon, Mar 26, 2018 at 11:38 PM, Fritz Budiyanto <[hidden email]> wrote:

Hi All,

Anyone know if Flink has TLS/SSL support for the current ES connector ?
If yes, any sample configuration/code ?
If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ?

Thanks,
Fritz