Kafka Connector Topic Discovery

[Martin, Nick-2]

I’m trying to use the topic discovery feature of the Kafka Connector. The problem I’m having is that Kafka Consumers fail to start if there are no topics matching the topic regex when they start up. Is this intended behavior? Is there some other property I could set to just continue discovery until they find a matching topic?

 

Background:

My application uses dynamically generated topic names where specific messages are sent on different topics based on some metadata in the messages. A producer service reads the metadata, determines the topic the data should be sent to, applies some validation logic to the topic name, creates that topic if it doesn’t already exist, and then sends the data. The problem is, when starting the whole stack with a fresh Kafka cluster, my Flink job with the Kafka consumer can’t start until the producer service has been started and has processed some data so that at least one matching topic exists. This sort of startup order dependency is obviously undesirable in a distributed microservice architecture.

 

Are there existing features/configuration settings that solve this problem? Should I open a ticket?

[Arvid Heise-4]

Hi Nick,

This looks like a valid use case and shouldn't fail. The only workaround I see is to create some dummy topics then.

On Fri, Jun 11, 2021 at 12:11 AM Martin, Nick J [US] (SP) <[hidden email]> wrote:

I’m trying to use the topic discovery feature of the Kafka Connector. The problem I’m having is that Kafka Consumers fail to start if there are no topics matching the topic regex when they start up. Is this intended behavior? Is there some other property I could set to just continue discovery until they find a matching topic?

 

Background:

My application uses dynamically generated topic names where specific messages are sent on different topics based on some metadata in the messages. A producer service reads the metadata, determines the topic the data should be sent to, applies some validation logic to the topic name, creates that topic if it doesn’t already exist, and then sends the data. The problem is, when starting the whole stack with a fresh Kafka cluster, my Flink job with the Kafka consumer can’t start until the producer service has been started and has processed some data so that at least one matching topic exists. This sort of startup order dependency is obviously undesirable in a distributed microservice architecture.

 

Are there existing features/configuration settings that solve this problem? Should I open a ticket?