Re: Using Kafka as bounded source with DataStream API in batch mode (Flink 1.12)

Posted by sagar on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Using-Kafka-as-bounded-source-with-DataStream-API-in-batch-mode-Flink-1-12-tp40637p40642.html

Hi Ardhani,

So whenever I want to run this flink job, I will call the Java API to put the data to the four different kafka topics, what data to put into kafka will be coded into those API and then once that is complete, I want to run the flink job on the available data in the kafka and perform business operation on all the available data.

I am not sure whether kafka as a datasource will be best for this use case, but somehow I don't want to expose my flink job to database tables directly. 

 

Thanks & Regards,

Sagar 

On Thu, Jan 14, 2021 at 12:41 PM Ardhani Narasimha <[hidden email]> wrote:

Interesting use case. 

Can you please elaborate more on this.

On what criteria do you want to batch? Time? Count? Or Size?

On Thu, 14 Jan 2021 at 12:15 PM, sagar <[hidden email]> wrote:

Hi Team,

I am getting the following error while running DataStream API in with batch mode with kafka source.

I am using FlinkKafkaConsumer to consume the data.

Caused by: java.lang.IllegalStateException: Detected an UNBOUNDED source with the 'execution.runtime-mode' set to 'BATCH'. This combination is not allowed, please set the 'execution.runtime-mode' to STREAMING or AUTOMATIC
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:198) ~[flink-core-1.12.0.jar:1.12.0]

In my batch program I wanted to work with four to five different stream in batch mode as data source is bounded

I don't find any clear example of how to do it with kafka souce with Flink 1.12

I don't want to use JDBC source as underlying database table may change. please give me some example on how to achieve the above use case.

Also for any large bounded source are there any alternatives to achieve this?

--

---Regards---

  Sagar Bandal

This is confidential mail ,All Rights are Reserved.If you are not intended receipiant please ignore this email.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

IMPORTANT: The contents of this email and any attachments are confidential and protected by applicable laws. If you have received this email by mistake, please (i) notify the sender immediately; (ii) delete it from your database; and (iii) do not disclose the contents to anyone or make copies thereof. Razorpay accepts no liability caused due to any inadvertent/ unintentional data transmitted through this email.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

--