Read file from S3 and write to kafka

[anurag]

Hi All,

I am searched a lot on google but could not find how I can achieve writing a flink  function which reads a file in S3 and for each line in the file write a message to kafka.

Thanks a lot , much appreciated. I am sorry if I did not searched properly.
Thanks,

Anurag

[miki haiat]

You can use the DataSet API to parse files from S3.

https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/#data-sources

https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#s3-simple-storage-service 

And then  parsed it and send it to kafka.

 

On Tue, Jun 4, 2019 at 5:57 AM anurag <[hidden email]> wrote:

Hi All,

I am searched a lot on google but could not find how I can achieve writing a flink  function which reads a file in S3 and for each line in the file write a message to kafka.

Thanks a lot , much appreciated. I am sorry if I did not searched properly.
Thanks,

Anurag