using regular expression to specify Kafka topics

[Soheil Pourbafrani]

I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.

Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.

Is there something equivalent or some other way to do that?

[Tzu-Li (Gordon) Tai]

Hi Soheil,

That feature is actually already internally available.

The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.

Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.

I’m not sure of exposing the pattern-based subscription API now for 1.4, though.

Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.

cc @Aljoscha, what do you think about this?

Best,

Gordon

On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani ([hidden email]) wrote:

I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.

Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.

Is there something equivalent or some other way to do that?

[Tzu-Li (Gordon) Tai]

Hi,

I’ve created a PR to publicly expose the feature: https://github.com/apache/flink/pull/5117.

Whether or not we should include this in the next release candidate for 1.4 is still up for discussion.

Best,

Gordon

On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

Hi Soheil,

That feature is actually already internally available.

The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.

Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.

I’m not sure of exposing the pattern-based subscription API now for 1.4, though.

Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.

cc @Aljoscha, what do you think about this?

Best,

Gordon

On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani ([hidden email]) wrote:

I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.

Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.

Is there something equivalent or some other way to do that?

[Gyula Fóra]

Hi,

I think what we need to ask here is whether the custom workaround solutions people are using over the place are any more robust than our own implementation.

I would be in favor of including it if the PR is merged in time for the next RC.

Gyula

Tzu-Li (Gordon) Tai <[hidden email]> ezt írta (időpont: 2017. dec. 4., H, 12:36):

Hi,

I’ve created a PR to publicly expose the feature: https://github.com/apache/flink/pull/5117.

Whether or not we should include this in the next release candidate for 1.4 is still up for discussion.

Best,

Gordon

On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

Hi Soheil,

That feature is actually already internally available.

The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.

Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.

I’m not sure of exposing the pattern-based subscription API now for 1.4, though.

Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.

cc @Aljoscha, what do you think about this?

Best,

Gordon

On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani ([hidden email]) wrote:

I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.

Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.

Is there something equivalent or some other way to do that?

[Aljoscha Krettek]

+1 for including in the next RC.

On 4. Dec 2017, at 13:10, Gyula Fóra <[hidden email]> wrote:

Hi,

I think what we need to ask here is whether the custom workaround solutions people are using over the place are any more robust than our own implementation.

I would be in favor of including it if the PR is merged in time for the next RC.

Gyula

Tzu-Li (Gordon) Tai <[hidden email]> ezt írta (időpont: 2017. dec. 4., H, 12:36):

Hi,

I’ve created a PR to publicly expose the feature: https://github.com/apache/flink/pull/5117.

Whether or not we should include this in the next release candidate for 1.4 is still up for discussion.

Best,

Gordon

On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

Hi Soheil,

That feature is actually already internally available.

The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.

Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.

I’m not sure of exposing the pattern-based subscription API now for 1.4, though.

Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.

cc @Aljoscha, what do you think about this?

Best,

Gordon

On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani ([hidden email]) wrote:

I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.

Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.

Is there something equivalent or some other way to do that?