[DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

[Stephan Ewen]

Hi all!

We still maintain connectors for Kafka 0.8 and 0.9 in Flink.

I would suggest to drop those with Flink 1.10 and start supporting only Kafka 0.10 onwards.

Are there any concerns about this, or still a significant number of users of these versions?

Best,

Stephan

[Wesley Peng-2]

on 2019/9/11 16:17, Stephan Ewen wrote:
> We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
> I would suggest to drop those with Flink 1.10 and start supporting only
> Kafka 0.10 onwards.
>
> Are there any concerns about this, or still a significant number of
> users of these versions?

But we still have a large scale of deployment kafka 0.9 in production.
Do you mean the new coming flink won't support kafka 0.9?
Though I understand for it, but just sounds sorry. :)

regards.

[Becket Qin]

Thanks for bringing this up, Stephan.

I am +1 on dropping support for Kafka 0.8. It is a pretty old version and I don't think there are many users on that version now.

However, for Kafka 0.9, I think there are still quite some users on that version. It might be better to keep it a little longer.

Given that dropping connector support is a backwards incompatible change, technically speaking we need to bump up the major version. I am not sure if release 1.10 is a good version to remove the code.

Personally I prefer to mark Kafka 0.8 connector as deprecated and remove the code base in Flink 2.x.

Thanks,

Jiangjie (Becket) Qin

On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng <[hidden email]> wrote:

on 2019/9/11 16:17, Stephan Ewen wrote:
> We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
> I would suggest to drop those with Flink 1.10 and start supporting only
> Kafka 0.10 onwards.
>
> Are there any concerns about this, or still a significant number of
> users of these versions?

But we still have a large scale of deployment kafka 0.9 in production.
Do you mean the new coming flink won't support kafka 0.9?
Though I understand for it, but just sounds sorry. :)

regards.

[Stephan Ewen]

I think this is very hypothetical, requiring a major version bump to drop an old connector version. What is the actual problem that would arise for users?

We have not required a major version to drop some old connectors that in the past, and if we want to continue "mono repo" style in Flink, I don't think we should have that as a constraint. It ties us to too much legacy code maintenance.

Best,

Stephan

On Mon, Sep 16, 2019 at 1:43 AM Becket Qin <[hidden email]> wrote:

Thanks for bringing this up, Stephan.

I am +1 on dropping support for Kafka 0.8. It is a pretty old version and I don't think there are many users on that version now.

However, for Kafka 0.9, I think there are still quite some users on that version. It might be better to keep it a little longer.

Given that dropping connector support is a backwards incompatible change, technically speaking we need to bump up the major version. I am not sure if release 1.10 is a good version to remove the code.

Personally I prefer to mark Kafka 0.8 connector as deprecated and remove the code base in Flink 2.x.

Thanks,

Jiangjie (Becket) Qin

On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng <[hidden email]> wrote:

on 2019/9/11 16:17, Stephan Ewen wrote:
> We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
> I would suggest to drop those with Flink 1.10 and start supporting only
> Kafka 0.10 onwards.
>
> Are there any concerns about this, or still a significant number of
> users of these versions?

But we still have a large scale of deployment kafka 0.9 in production.
Do you mean the new coming flink won't support kafka 0.9?
Though I understand for it, but just sounds sorry. :)

regards.

[Till Rohrmann]

I second Stephan here. Moreover, given that the source and sink interfaces are public API, it should still be possible to take the Kafka 0.8/0.9 connector from a previous Flink version and run it with a newer version.

Cheers,

Till

On Mon, Sep 16, 2019 at 10:06 AM Stephan Ewen <[hidden email]> wrote:

I think this is very hypothetical, requiring a major version bump to drop an old connector version. What is the actual problem that would arise for users?

We have not required a major version to drop some old connectors that in the past, and if we want to continue "mono repo" style in Flink, I don't think we should have that as a constraint. It ties us to too much legacy code maintenance.

Best,

Stephan

On Mon, Sep 16, 2019 at 1:43 AM Becket Qin <[hidden email]> wrote:

Thanks for bringing this up, Stephan.

I am +1 on dropping support for Kafka 0.8. It is a pretty old version and I don't think there are many users on that version now.

However, for Kafka 0.9, I think there are still quite some users on that version. It might be better to keep it a little longer.

Given that dropping connector support is a backwards incompatible change, technically speaking we need to bump up the major version. I am not sure if release 1.10 is a good version to remove the code.

Personally I prefer to mark Kafka 0.8 connector as deprecated and remove the code base in Flink 2.x.

Thanks,

Jiangjie (Becket) Qin

On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng <[hidden email]> wrote:

on 2019/9/11 16:17, Stephan Ewen wrote:
> We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
> I would suggest to drop those with Flink 1.10 and start supporting only
> Kafka 0.10 onwards.
>
> Are there any concerns about this, or still a significant number of
> users of these versions?

But we still have a large scale of deployment kafka 0.9 in production.
Do you mean the new coming flink won't support kafka 0.9?
Though I understand for it, but just sounds sorry. :)

regards.

[Aljoscha Krettek]

Hi,

I’m also in favour of at least dropping support for Kafka 0.8.

More generally, I think we have to be careful when recommending to use Flink connectors from older versions with a newer Flink versions. The interfaces might be stable but they might use internal code that is not stable, as shown by [1].

Best,
Aljoscha

[1] https://issues.apache.org/jira/browse/FLINK-13586

> On 16. Sep 2019, at 11:09, Till Rohrmann <[hidden email]> wrote:
>
> I second Stephan here. Moreover, given that the source and sink interfaces are public API, it should still be possible to take the Kafka 0.8/0.9 connector from a previous Flink version and run it with a newer version.
>
> Cheers,
> Till
>
> On Mon, Sep 16, 2019 at 10:06 AM Stephan Ewen <[hidden email]> wrote:
> I think this is very hypothetical, requiring a major version bump to drop an old connector version. What is the actual problem that would arise for users?
>
> We have not required a major version to drop some old connectors that in the past, and if we want to continue "mono repo" style in Flink, I don't think we should have that as a constraint. It ties us to too much legacy code maintenance.
>
> Best,
> Stephan
>
>
> On Mon, Sep 16, 2019 at 1:43 AM Becket Qin <[hidden email]> wrote:
> Thanks for bringing this up, Stephan.
>
> I am +1 on dropping support for Kafka 0.8. It is a pretty old version and I don't think there are many users on that version now.
>
> However, for Kafka 0.9, I think there are still quite some users on that version. It might be better to keep it a little longer.
>
> Given that dropping connector support is a backwards incompatible change, technically speaking we need to bump up the major version. I am not sure if release 1.10 is a good version to remove the code.
> Personally I prefer to mark Kafka 0.8 connector as deprecated and remove the code base in Flink 2.x.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Wed, Sep 11, 2019 at 4:26 PM Wesley Peng <[hidden email]> wrote:
>
>
> on 2019/9/11 16:17, Stephan Ewen wrote:
> > We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
> > I would suggest to drop those with Flink 1.10 and start supporting only
> > Kafka 0.10 onwards.
> >
> > Are there any concerns about this, or still a significant number of
> > users of these versions?
>
> But we still have a large scale of deployment kafka 0.9 in production.
> Do you mean the new coming flink won't support kafka 0.9?
> Though I understand for it, but just sounds sorry. :)
>
> regards.