Re: Local Cluster have problem with connect to elasticsearch

Posted by Tzu-Li Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Local-Cluster-have-problem-with-connect-to-elasticsearch-tp6788p6838.html

Hi Rafal,

From your description, it seems like Flink is complaining because it cannot access the Elasticsearch API related dependencies as well. You'd also have to include the following into your Maven build, under <artifactItems>:

<artifactItem>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.3.2</version>
    <type>jar</type>
    <overWrite>false</overWrite>
    <outputDirectory>${project.build.directory}/classes</outputDirectory>
    <includes>org/elasticsearch/**</includes>
</artifactItem>

Now your built jar should correctly include all required dependencies (the connector & Elasticsearch API).

As explained in Linking with modules not contained in the binary distribution, it will be enough to package dependencies along with your code for Flink to access all required dependencies, and you wouldn't need to copy the jar to the lib folder. I would recommend to clean up the lib folder of the previous jars you copied, and follow this approach in the future, just in case they mess up the classloader.

As with your first attempt that Flink cannot find any Elasticsearch nodes when executed in the IDE, I'm suspecting the reason is that the elasticsearch2 connector by default uses version 2.2.1, lower than your cluster version 2.3.2. I had previous experience when Elasticsearch strangely complains not finding any nodes when using lower client versions than the deployment. Can you try compiling the elasticsearch2 connector with the option -Delasticsearch.version=2.3.2, and use the newly build connector jar, following the same method mentioned above?

Hope this helps!

Cheers,
Gordon