A strange exception while consumption using a multi topic Kafka Connector

[Vishal Santoshi]

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)

.

.

.

This flink 1.4.0

Any ideas folks ?

[Vishal Santoshi]

??

On Fri, May 11, 2018, 8:15 PM Vishal Santoshi <[hidden email]> wrote:

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)

.

.

.

This flink 1.4.0

Any ideas folks ?

[Ted Yu]

I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

It seems the List subscribedPartitionStates was being modified when runFetchLoop iterated the List.

This can happen if, e.g., FlinkKafkaConsumer runs the following code concurrently:

                kafkaFetcher.addDiscoveredPartitions(discoveredPartitions);

You can log a JIRA.

If you have unit test reproducing this, that would be great.

FYI

On Fri, May 11, 2018 at 5:15 PM, Vishal Santoshi <[hidden email]> wrote:

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)

.

.

.

This flink 1.4.0

Any ideas folks ?

[Ted Yu]

FLINK-9349 was logged.

FYI

On Sat, May 12, 2018 at 7:52 AM, Ted Yu <[hidden email]> wrote:

I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

It seems the List subscribedPartitionStates was being modified when runFetchLoop iterated the List.

This can happen if, e.g., FlinkKafkaConsumer runs the following code concurrently:

                kafkaFetcher.addDiscoveredPartitions(discoveredPartitions);

You can log a JIRA.

If you have unit test reproducing this, that would be great.

FYI

On Fri, May 11, 2018 at 5:15 PM, Vishal Santoshi <[hidden email]> wrote:

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)

.

.

.

This flink 1.4.0

Any ideas folks ?

[Vishal Santoshi]

Yep, Thanks. We have a set up where topics are constantly being added on a kakfa-to-hdfs pipeline. 

On Sun, May 13, 2018 at 11:58 AM, Ted Yu <[hidden email]> wrote:

FLINK-9349 was logged.

FYI

On Sat, May 12, 2018 at 7:52 AM, Ted Yu <[hidden email]> wrote:

I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

It seems the List subscribedPartitionStates was being modified when runFetchLoop iterated the List.

This can happen if, e.g., FlinkKafkaConsumer runs the following code concurrently:

                kafkaFetcher.addDiscoveredPartitions(discoveredPartitions);

You can log a JIRA.

If you have unit test reproducing this, that would be great.

FYI

On Fri, May 11, 2018 at 5:15 PM, Vishal Santoshi <[hidden email]> wrote:

java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)

.

.

.

This flink 1.4.0

Any ideas folks ?