PyFlink DataStream Example Kafka/Kinesis?

[kb]

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[kb]

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[kb]

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[Shuiqiang Chen]

Hi Kevin,

Kinesis connector is not supported yet in Python DataStream API. We will add it in the future.

Best,

Shuiqiang

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 上午5:03写道：

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[Shuiqiang Chen]

I have just created the jira https://issues.apache.org/jira/browse/FLINK-21966 and will finish it soon.

Best,

Shuiqiang

Xinbin Huang <[hidden email]> 于2021年3月25日周四 上午10:43写道：

Hi Shuiqiang,

I am interested in the same feature. Do we have a ticket to track this right now?

Best

Bin

On Wed, Mar 24, 2021 at 7:30 PM Shuiqiang Chen <[hidden email]> wrote:

Hi Kevin,

Kinesis connector is not supported yet in Python DataStream API. We will add it in the future.

Best,

Shuiqiang

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 上午5:03写道：

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[Shuiqiang Chen]

Sure, no problem.  You can refer to the implementation of Kafka connector, they are very much alike. 

Xinbin Huang <[hidden email]> 于2021年3月25日周四 上午10:55写道：

Hi Shuiqiang,

Thanks for the quick response on creating the ticket for Kinesis Connector. Do you mind giving me the chance to try to implement the connector over the weekend?

I am interested in contributing to Flink, and I think this can be a good starting point to me

Best

Bin

On Wed, Mar 24, 2021 at 7:49 PM Shuiqiang Chen <[hidden email]> wrote:

I have just created the jira https://issues.apache.org/jira/browse/FLINK-21966 and will finish it soon.

Best,

Shuiqiang

Xinbin Huang <[hidden email]> 于2021年3月25日周四 上午10:43写道：

Hi Shuiqiang,

I am interested in the same feature. Do we have a ticket to track this right now?

Best

Bin

On Wed, Mar 24, 2021 at 7:30 PM Shuiqiang Chen <[hidden email]> wrote:

Hi Kevin,

Kinesis connector is not supported yet in Python DataStream API. We will add it in the future.

Best,

Shuiqiang

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 上午5:03写道：

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[kb]

Hi Shuiqiang,

 

Thanks for letting me know. Feel free to send any beginner level contributions for this effort my way 😊 .

 

Best,

kevin

 

From: Shuiqiang Chen <[hidden email]>
Date: Wednesday, March 24, 2021 at 10:31 PM
To: "Bohinski, Kevin" <[hidden email]>
Cc: user <[hidden email]>
Subject: [EXTERNAL] Re: PyFlink DataStream Example Kafka/Kinesis?

 

Hi Kevin,

 

Kinesis connector is not supported yet in Python DataStream API. We will add it in the future.

 

Best,

Shuiqiang

 

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 上午5:03写道：

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin

[Shuiqiang Chen]

Hi Kevin, Xinbin,

Hi Shuiqiang,

Thanks for the quick response on creating the ticket for Kinesis Connector. Do you mind giving me the chance to try to implement the connector over the weekend?

I am interested in contributing to Flink, and I think this can be a good starting point to me

Best

Bin

Sorry for the confusion. Xinbin had personal letter that could not be displayed in this email thread that he is interested in solving this issue https://issues.apache.org/jira/browse/FLINK-21966.  So both of you can participate in this contribution as you like.

Best,

Shuiqiang

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 下午12:00写道：

Hi Shuiqiang,

 

Thanks for letting me know. Feel free to send any beginner level contributions for this effort my way 😊 .

 

Best,

kevin

 

From: Shuiqiang Chen <[hidden email]>
Date: Wednesday, March 24, 2021 at 10:31 PM
To: "Bohinski, Kevin" <[hidden email]>
Cc: user <[hidden email]>
Subject: [EXTERNAL] Re: PyFlink DataStream Example Kafka/Kinesis?

 

Hi Kevin,

 

Kinesis connector is not supported yet in Python DataStream API. We will add it in the future.

 

Best,

Shuiqiang

 

Bohinski, Kevin <[hidden email]> 于2021年3月25日周四 上午5:03写道：

Is there a kinesis example?

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:40 PM
To: "Bohinski, Kevin" <[hidden email]>
Subject: Re: PyFlink DataStream Example Kafka/Kinesis?

 

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py

 

From: "Bohinski, Kevin" <[hidden email]>
Date: Wednesday, March 24, 2021 at 4:38 PM
To: user <[hidden email]>
Subject: PyFlink DataStream Example Kafka/Kinesis?

 

Hi,

 

Is there an example kafka/kinesis source or sink for the PyFlink DataStream API?

 

Best,

kevin