Flink Kafka Unused Properties Log Warnings

[David Kim]

Hello!

Using Flink 1.0.3.

This is cosmetic but will help clean up logging I think.

The Apache KafkaConsumer logs a warning [1] for any unused properties. This is great in case the developer has a typo or should clean up any unused keys.

Flink's Kafka consumer and producer have some custom properties and it passes the entire properties map into the Apache Kafka consumer and producer. Custom properties such as "flink.poll-timeout" make it through and it triggers a warning.

For the sake of cleaning up logging and removing false positives, I think it might make sense for the FlinkKafkaConsumer09 [2] and FlinkKafkaProducerBase [3] to filter out custom flink properties.

Let me know if this sounds reasonable!

Thanks!

David

[1] https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639

[2] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186

[3] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153

[rmetzger0]

Hi Kim,

I agree with you. We should definitively filter out custom Flink properties. I've filed a JIRA for this: https://issues.apache.org/jira/browse/FLINK-4004

Thanks for the report!

Regards,

Robert

On Wed, Jun 1, 2016 at 6:42 PM, David Kim <[hidden email]> wrote:

Hello!

Using Flink 1.0.3.

This is cosmetic but will help clean up logging I think.

The Apache KafkaConsumer logs a warning [1] for any unused properties. This is great in case the developer has a typo or should clean up any unused keys.

Flink's Kafka consumer and producer have some custom properties and it passes the entire properties map into the Apache Kafka consumer and producer. Custom properties such as "flink.poll-timeout" make it through and it triggers a warning.

For the sake of cleaning up logging and removing false positives, I think it might make sense for the FlinkKafkaConsumer09 [2] and FlinkKafkaProducerBase [3] to filter out custom flink properties.

Let me know if this sounds reasonable!

Thanks!

David

[1] https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639

[2] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186

[3] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153

[David Kim]

Awesome, thanks Robert!

On Wed, Jun 1, 2016 at 1:12 PM Robert Metzger <[hidden email]> wrote:

Hi Kim,

I agree with you. We should definitively filter out custom Flink properties. I've filed a JIRA for this: https://issues.apache.org/jira/browse/FLINK-4004

Thanks for the report!

Regards,

Robert

On Wed, Jun 1, 2016 at 6:42 PM, David Kim <[hidden email]> wrote:

Hello!

Using Flink 1.0.3.

This is cosmetic but will help clean up logging I think.

The Apache KafkaConsumer logs a warning [1] for any unused properties. This is great in case the developer has a typo or should clean up any unused keys.

Flink's Kafka consumer and producer have some custom properties and it passes the entire properties map into the Apache Kafka consumer and producer. Custom properties such as "flink.poll-timeout" make it through and it triggers a warning.

For the sake of cleaning up logging and removing false positives, I think it might make sense for the FlinkKafkaConsumer09 [2] and FlinkKafkaProducerBase [3] to filter out custom flink properties.

Let me know if this sounds reasonable!

Thanks!

David

[1] https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639

[2] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186

[3] https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153