How To subscribe a Kinesis Stream using enhance fanout?

[Xiaolong Wang]

Hello Flink Community!

      I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.

     But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.

     Has this issue occurred to anyone before?

    Thanks for your help.

[Tzu-Li (Gordon) Tai]

Hi Xiaolong,

You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1].
Could you open a JIRA ticket for this?
As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <[hidden email]> wrote:

Hello Flink Community!

      I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.

     But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.

     Has this issue occurred to anyone before?

    Thanks for your help.

[Xiaolong Wang]

Thanks, I'll check it out. 

On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Xiaolong,

You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1].
Could you open a JIRA ticket for this?
As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <[hidden email]> wrote:

Hello Flink Community!

      I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.

     But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.

     Has this issue occurred to anyone before?

    Thanks for your help.

[orionemail]

Hi,

We also recently needed this functionality, unfortunately we were unable to implement it ourselves so changed our plan accordingly.

However we very much see the benefit for this feature and would be interested in following the JIRA ticket.

Thanks

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, 14 May 2020 11:34, Xiaolong Wang <[hidden email]> wrote:

Thanks, I'll check it out. 

On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Xiaolong,

You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1].

Could you open a JIRA ticket for this?

As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers.

Cheers,

Gordon

[1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <[hidden email]> wrote:

Hello Flink Community!

      I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.

     But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.

     Has this issue occurred to anyone before?

    Thanks for your help.

[Tzu-Li (Gordon) Tai]

Hi all,

Just to give a quick update: there will be contributors from the AWS Kinesis team working on contributing enhanced fan out support to the connector.
You can follow the progress here: https://issues.apache.org/jira/browse/FLINK-17688

Cheers,
Gordon

On Fri, May 15, 2020 at 5:55 PM orionemail <[hidden email]> wrote:

Hi,

We also recently needed this functionality, unfortunately we were unable to implement it ourselves so changed our plan accordingly.

However we very much see the benefit for this feature and would be interested in following the JIRA ticket.

Thanks

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, 14 May 2020 11:34, Xiaolong Wang <[hidden email]> wrote:

Thanks, I'll check it out. 

On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Xiaolong,

You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1].

Could you open a JIRA ticket for this?

As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers.

Cheers,

Gordon

[1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang <[hidden email]> wrote:

Hello Flink Community!

      I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message.

     But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations.

     Has this issue occurred to anyone before?

    Thanks for your help.