Flink Elastic Sink AWS ES

[ant burton]

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[Ted Yu]

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[ant burton]

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[arpit8622]

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[ant burton]

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[Vinay Patil]

Hi,

We have recently moved to AWS ES service, I am using the following code:

https://github.com/awslabs/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/src/main/java/com/amazonaws/flink/refarch/utils/ElasticsearchJestSink.java

(Note that this is not the inbuilt Flink ESSink)

You can read this blog post:

https://aws.amazon.com/blogs/big-data/build-a-real-time-stream-processing-pipeline-with-apache-flink-on-aws/

On Sun, Aug 27, 2017 at 7:02 PM, ant burton [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Elastic-Sink-AWS-ES-tp15162p15173.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[ant burton]

Hi Arpit,

The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced?

curl -XGET 'http://<your aws es url>/_nodes'

Thanks,

On 27 Aug 2017, at 14:32, ant burton <[hidden email]> wrote:

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[arpit8622]

Hi Ant,

Can you try this.

curl -XGET 'http://<your aws es url>/_cat/nodes?v&h=ip,port'

This should give you ip and port

On Mon, Aug 28, 2017 at 3:42 AM, ant burton <[hidden email]> wrote:

Hi Arpit,

The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced?

curl -XGET 'http://<your aws es url>/_nodes'

Thanks,

On 27 Aug 2017, at 14:32, ant burton <[hidden email]> wrote:

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[ant burton]

Hey Arpit,

_cat/nodes?v&h=ip,port

returns the following which I have not added the x’s they were returned on the response

ip port

x.x.x.x 9300

Thanks your for you help

Anthony

On 28 Aug 2017, at 10:34, arpit srivastava <[hidden email]> wrote:

Hi Ant,

Can you try this.

curl -XGET 'http://<your aws es url>/_cat/nodes?v&h=ip,port'

This should give you ip and port

On Mon, Aug 28, 2017 at 3:42 AM, ant burton <[hidden email]> wrote:

Hi Arpit,

The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced?

curl -XGET 'http://<your aws es url>/_nodes'

Thanks,

On 27 Aug 2017, at 14:32, ant burton <[hidden email]> wrote:

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,

[arpit8622]

It seems AWS ES setup is hiding the nodes ip.

Then I think you can try @vinay patil's solution.

Thanks,

Arpit

On Tue, Aug 29, 2017 at 3:56 AM, ant burton <[hidden email]> wrote:

Hey Arpit,

_cat/nodes?v&h=ip,port

returns the following which I have not added the x’s they were returned on the response

ip port

x.x.x.x 9300

Thanks your for you help

Anthony

On 28 Aug 2017, at 10:34, arpit srivastava <[hidden email]> wrote:

Hi Ant,

Can you try this.

curl -XGET 'http://<your aws es url>/_cat/nodes?v&h=ip,port'

This should give you ip and port

On Mon, Aug 28, 2017 at 3:42 AM, ant burton <[hidden email]> wrote:

Hi Arpit,

The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced?

curl -XGET 'http://<your aws es url>/_nodes'

Thanks,

On 27 Aug 2017, at 14:32, ant burton <[hidden email]> wrote:

Thanks! I'll check later this evening.

On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email]> wrote:

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port.

For using Flink we got the actual ip address of the ES nodes and put that in ips below.

transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))

But this worked only because 9300 port was open on ES nodes in our setup and so accessible from our Flink cluster.​

Get your node list on your ES Cluster using

curl -XGET 'http://<your aws es url>/_nodes'

​and then check whether you can telnet on that <es node ip> on port 9300 from your flink cluster nodes

$ telnet <es node ip> 9300

If this works then you can use above solution.​

On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email]> wrote:

Hi Ted,

Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message

my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using port 9300 in a docker environment locally. 

Thanks,

On 26 Aug 2017, at 23:24, Ted Yu <[hidden email]> wrote:

If port 9300 in the following example is replaced by 9200, would that work ?

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html

Please use Flink 1.3.1+

On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email]> wrote:

Hello,

Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES.

I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector.

The error message that occurs when attempting to connect to AWS ES via port 80 (9200) with the Flink Elasticsearch connector is:

    Elasticsearch client is not connected to any Elasticsearch nodes!

Could anybody confirm the above? and if possible provide an alternative solution?

Thanks you,