Elasticsearch connector and number of shards

[Flavio Pompermaier]

Hi to all,

we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we wanted to set the number of shards when creating a new index. Is that possible at the moment?

Best,

Flavio

[Ufuk Celebi]

I've never used the Elasticsearch sink, but the docs say:

"Note how a Map of Strings is used to configure the Sink. The
configuration keys are documented in the Elasticsearch
documentationhere. Especially important is the cluster.name parameter
that must correspond to the name of your cluster."

The config keys for index creation are found here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

Does this help? If not, maybe @Aljoscha can chime in here.

– Ufuk


On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
<[hidden email]> wrote:
> Hi to all,
>
> we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
> wanted to set the number of shards when creating a new index. Is that
> possible at the moment?
>
> Best,
> Flavio

[Flavio Pompermaier]

Indeed, we've tried with the parameter index.number_of_shards but it didn't work so I fear that this parameter is not handled by the current implementation..am I wrong?

On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <[hidden email]> wrote:

I've never used the Elasticsearch sink, but the docs say:

"Note how a Map of Strings is used to configure the Sink. The
configuration keys are documented in the Elasticsearch
documentationhere. Especially important is the cluster.name parameter
that must correspond to the name of your cluster."

The config keys for index creation are found here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

Does this help? If not, maybe @Aljoscha can chime in here.

– Ufuk

On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
<[hidden email]> wrote:
> Hi to all,
>
> we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
> wanted to set the number of shards when creating a new index. Is that
> possible at the moment?
>
> Best,
> Flavio

[Maximilian Michels]

The connector doesn't cover this use case. Through the API you need to
use the IndicesAdminClient:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html

Otherwise Elasticsearch creates an index with shards automatically. We
could add support for configuring shards in the future.

On Mon, Jul 18, 2016 at 11:42 AM, Flavio Pompermaier
<[hidden email]> wrote:

> Indeed, we've tried with the parameter index.number_of_shards but it didn't
> work so I fear that this parameter is not handled by the current
> implementation..am I wrong?
>
> On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <[hidden email]> wrote:
>>
>> I've never used the Elasticsearch sink, but the docs say:
>>
>> "Note how a Map of Strings is used to configure the Sink. The
>> configuration keys are documented in the Elasticsearch
>> documentationhere. Especially important is the cluster.name parameter
>> that must correspond to the name of your cluster."
>>
>> The config keys for index creation are found here:
>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
>>
>> Does this help? If not, maybe @Aljoscha can chime in here.
>>
>> – Ufuk
>>
>>
>> On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
>> <[hidden email]> wrote:
>> > Hi to all,
>> >
>> > we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
>> > wanted to set the number of shards when creating a new index. Is that
>> > possible at the moment?
>> >
>> > Best,
>> > Flavio
>
>

[Flavio Pompermaier]

I've just added a JIRA improvement ticket for this (https://issues.apache.org/jira/browse/FLINK-4491).

Best,

Flavio

On Wed, Jul 20, 2016 at 4:21 PM, Maximilian Michels <[hidden email]> wrote:

The connector doesn't cover this use case. Through the API you need to
use the IndicesAdminClient:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html

Otherwise Elasticsearch creates an index with shards automatically. We
could add support for configuring shards in the future.

On Mon, Jul 18, 2016 at 11:42 AM, Flavio Pompermaier

<[hidden email]> wrote:
> Indeed, we've tried with the parameter index.number_of_shards but it didn't
> work so I fear that this parameter is not handled by the current
> implementation..am I wrong?
>
> On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <[hidden email]> wrote:
>>
>> I've never used the Elasticsearch sink, but the docs say:
>>
>> "Note how a Map of Strings is used to configure the Sink. The
>> configuration keys are documented in the Elasticsearch
>> documentationhere. Especially important is the cluster.name parameter
>> that must correspond to the name of your cluster."
>>
>> The config keys for index creation are found here:
>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
>>
>> Does this help? If not, maybe @Aljoscha can chime in here.
>>
>> – Ufuk
>>
>>
>> On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
>> <[hidden email]> wrote:
>> > Hi to all,
>> >
>> > we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
>> > wanted to set the number of shards when creating a new index. Is that
>> > possible at the moment?
>> >
>> > Best,
>> > Flavio
>
>