Re: FlinkKafkaConsumer09 not reading from all partitions?

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/FlinkKafkaConsumer09-not-reading-from-all-partitions-tp9096p9099.html

Is this based on log messages, or can you confirm that from some partitions, no data is read?

On Tue, Sep 20, 2016 at 6:03 PM, Curtis Wilde <[hidden email]> wrote:

I’m just getting my feet wet with Flink, doing a quick implementation on my laptop following the examples.

 

I’m consuming from a Kafka 0.9 cluster using FlinkKafkaConsumer09 from a topic with 50 partitions.

 

I see offset commit messages for partitions 0-9, but I don’t see offsets being committed for partitions 10-49.

 

Any ideas what’s going on?

 

I’ve been messing around with consumer configs to no avail:

 

enable.auto.commit=true,

group.id=flink-test,

flink.poll-timeout=9223372036854775807,

auto.offset.reset=earliest,

session.timeout.ms=120000,

bootstrap.servers=n1:9092,n2:9092,n3:9092,n4:9092,n5:9092,

request.timeout.ms=120500,

topic=mytopic,

max.partition.fetch.bytes=13631488,

auto.commit.interval.ms=5000