Get current kafka offsets for kafka sources

[Rex Fenley]

Hi,

Is there any way to fetch the current kafka topic offsets for the kafka sources for flink?

Thanks!

--

[Aeden Jameson]

My understanding is the FlinkKafkaConsumer is a wrapper around the
Kafka consumer libraries so if you've set the group.id property you
should be able to see the offsets with something like
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--group my-flink-application.

On Tue, Dec 15, 2020 at 9:39 PM Rex Fenley <[hidden email]> wrote:

>
> Hi,
>
> Is there any way to fetch the current kafka topic offsets for the kafka sources for flink?
>
> Thanks!
>
> --
>
> Rex Fenley  |  Software Engineer - Mobile and Backend
>
>
> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--
Cheers,
Aeden

GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Blah Blah Blah: http://www.twitter.com/daliful

[Rex Fenley]

I'll give a look into that approach. Thanks

On Tue, Dec 15, 2020 at 9:48 PM Aeden Jameson <[hidden email]> wrote:

My understanding is the FlinkKafkaConsumer is a wrapper around the
Kafka consumer libraries so if you've set the group.id property you
should be able to see the offsets with something like
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--group my-flink-application.

On Tue, Dec 15, 2020 at 9:39 PM Rex Fenley <[hidden email]> wrote:
>
> Hi,
>
> Is there any way to fetch the current kafka topic offsets for the kafka sources for flink?
>
> Thanks!
>
> --
>
> Rex Fenley  |  Software Engineer - Mobile and Backend
>
>
> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US



--
Cheers,
Aeden

GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Blah Blah Blah: http://www.twitter.com/daliful

--

[Juha Mynttinen-2]

Hey,

Have a look at [1]. Basically, you won't see the "real-time" consumer group offsets stored in Kafka itself, but only the ones the Flink Kafka consumer stores there when checkpointing (assuming you have checkpointing enabled). The same information is available in Flink metrics [2], "committedOffsets". I'm not sure, but I think "currentOffsets" could be the real-time offset, what you're probably looking for.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/metrics.html#kafka-connectors

Regards,

Juha

El mié, 16 dic 2020 a las 8:22, Rex Fenley (<[hidden email]>) escribió:

I'll give a look into that approach. Thanks

On Tue, Dec 15, 2020 at 9:48 PM Aeden Jameson <[hidden email]> wrote:

My understanding is the FlinkKafkaConsumer is a wrapper around the
Kafka consumer libraries so if you've set the group.id property you
should be able to see the offsets with something like
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--group my-flink-application.

On Tue, Dec 15, 2020 at 9:39 PM Rex Fenley <[hidden email]> wrote:
>
> Hi,
>
> Is there any way to fetch the current kafka topic offsets for the kafka sources for flink?
>
> Thanks!
>
> --
>
> Rex Fenley  |  Software Engineer - Mobile and Backend
>
>
> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US



--
Cheers,
Aeden

GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Blah Blah Blah: http://www.twitter.com/daliful

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US