Availability for the ElasticSearch 2 streaming connector

[Vieru, Mihail]

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail

[rmetzger0]

Hi Mihail,

It seems that nobody is actively working on the elasticsearch2 connector right now. The 1.0.0 release is already feature frozen, only bug fixes or (some) pending pull requests go in.

What you can always do is copy the code from our current elasticsearch connector, set the dependency to the version you would like to use and adopt our code to their API changes. I think it might take not much time to get it working.

(The reason why we usually need more time for stuff like this are integration tests and documentation).

Please let me know if that solution doesn't work for you.

Regards,

Robert

On Tue, Feb 16, 2016 at 2:53 PM, Vieru, Mihail <[hidden email]> wrote:

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail

[Suneel Marthi-2]

Hey I missed this thread, sorry about that.

I have a basic connector working with ES 2.0 which I can push out.  Its not optimized yet and I don't have the time to look at it, if someone would like to take it over go ahead I can send a PR.

On Wed, Feb 17, 2016 at 4:57 PM, Robert Metzger <[hidden email]> wrote:

Hi Mihail,

It seems that nobody is actively working on the elasticsearch2 connector right now. The 1.0.0 release is already feature frozen, only bug fixes or (some) pending pull requests go in.

What you can always do is copy the code from our current elasticsearch connector, set the dependency to the version you would like to use and adopt our code to their API changes. I think it might take not much time to get it working.

(The reason why we usually need more time for stuff like this are integration tests and documentation).

Please let me know if that solution doesn't work for you.

Regards,

Robert

On Tue, Feb 16, 2016 at 2:53 PM, Vieru, Mihail <[hidden email]> wrote:

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail

[Zach Cox]

I recently did exactly what Robert described: I copied the code from this (closed) PR https://github.com/apache/flink/pull/1479, modified it a bit, and just included it in my own project that uses the Elasticsearch 2 java api. Seems to work well. Here are the files so you can do the same:

https://gist.github.com/zcox/59e486be7aeeca381be0

-Zach

On Wed, Feb 17, 2016 at 4:06 PM Suneel Marthi <[hidden email]> wrote:

Hey I missed this thread, sorry about that.

I have a basic connector working with ES 2.0 which I can push out.  Its not optimized yet and I don't have the time to look at it, if someone would like to take it over go ahead I can send a PR.

On Wed, Feb 17, 2016 at 4:57 PM, Robert Metzger <[hidden email]> wrote:

Hi Mihail,

It seems that nobody is actively working on the elasticsearch2 connector right now. The 1.0.0 release is already feature frozen, only bug fixes or (some) pending pull requests go in.

What you can always do is copy the code from our current elasticsearch connector, set the dependency to the version you would like to use and adopt our code to their API changes. I think it might take not much time to get it working.

(The reason why we usually need more time for stuff like this are integration tests and documentation).

Please let me know if that solution doesn't work for you.

Regards,

Robert

On Tue, Feb 16, 2016 at 2:53 PM, Vieru, Mihail <[hidden email]> wrote:

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail

[Suneel Marthi]

Thanks Zach, I have a few minor changes too locally; I'll push a PR out tomorrow that has ur changes too. 

On Wed, Feb 17, 2016 at 5:13 PM, Zach Cox <[hidden email]> wrote:

I recently did exactly what Robert described: I copied the code from this (closed) PR https://github.com/apache/flink/pull/1479, modified it a bit, and just included it in my own project that uses the Elasticsearch 2 java api. Seems to work well. Here are the files so you can do the same:

https://gist.github.com/zcox/59e486be7aeeca381be0

-Zach

On Wed, Feb 17, 2016 at 4:06 PM Suneel Marthi <[hidden email]> wrote:

Hey I missed this thread, sorry about that.

I have a basic connector working with ES 2.0 which I can push out.  Its not optimized yet and I don't have the time to look at it, if someone would like to take it over go ahead I can send a PR.

On Wed, Feb 17, 2016 at 4:57 PM, Robert Metzger <[hidden email]> wrote:

Hi Mihail,

It seems that nobody is actively working on the elasticsearch2 connector right now. The 1.0.0 release is already feature frozen, only bug fixes or (some) pending pull requests go in.

What you can always do is copy the code from our current elasticsearch connector, set the dependency to the version you would like to use and adopt our code to their API changes. I think it might take not much time to get it working.

(The reason why we usually need more time for stuff like this are integration tests and documentation).

Please let me know if that solution doesn't work for you.

Regards,

Robert

On Tue, Feb 16, 2016 at 2:53 PM, Vieru, Mihail <[hidden email]> wrote:

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail

[Zach Cox]

Awesome, thanks Suneel. :D

I made the changes to support our use case, which needed flatMap behavior (index 2 docs, or zero docs, per incoming element) instead of map, and we also need to make either IndexRequest or UpdateRequest depending on the element.

-Zach

On Thu, Feb 18, 2016 at 2:06 AM Suneel Marthi <[hidden email]> wrote:

Thanks Zach, I have a few minor changes too locally; I'll push a PR out tomorrow that has ur changes too. 

On Wed, Feb 17, 2016 at 5:13 PM, Zach Cox <[hidden email]> wrote:

I recently did exactly what Robert described: I copied the code from this (closed) PR https://github.com/apache/flink/pull/1479, modified it a bit, and just included it in my own project that uses the Elasticsearch 2 java api. Seems to work well. Here are the files so you can do the same:

https://gist.github.com/zcox/59e486be7aeeca381be0

-Zach

On Wed, Feb 17, 2016 at 4:06 PM Suneel Marthi <[hidden email]> wrote:

Hey I missed this thread, sorry about that.

I have a basic connector working with ES 2.0 which I can push out.  Its not optimized yet and I don't have the time to look at it, if someone would like to take it over go ahead I can send a PR.

On Wed, Feb 17, 2016 at 4:57 PM, Robert Metzger <[hidden email]> wrote:

Hi Mihail,

It seems that nobody is actively working on the elasticsearch2 connector right now. The 1.0.0 release is already feature frozen, only bug fixes or (some) pending pull requests go in.

What you can always do is copy the code from our current elasticsearch connector, set the dependency to the version you would like to use and adopt our code to their API changes. I think it might take not much time to get it working.

(The reason why we usually need more time for stuff like this are integration tests and documentation).

Please let me know if that solution doesn't work for you.

Regards,

Robert

On Tue, Feb 16, 2016 at 2:53 PM, Vieru, Mihail <[hidden email]> wrote:

Hi,

in reference to this ticket https://issues.apache.org/jira/browse/FLINK-3115 when do you think that an ElasticSearch 2 streaming connector will become available? Will it make it for the 1.0 release?

That would be great, as we are planning to use that particular version of ElasticSearch in the very near future.

Best regards,

Mihail