Production Readiness of File Source

[chiggi_dev]

Hi,

I am intending to use the File source for a production use case. I have a few use cases that are currently not supported like deleting a file once it's processed. 

So I was wondering if we can use this in production or write my own implementation? Is there any recommendations around this?

Thanks

Chirag

[Dawid Wysakowicz-2]

Hi,

As for the issue of production readiness of the File Source(and other components) I'd recommend having a look at the PR, which is close to being merged where we express our opinion how we see certain components: https://github.com/apache/flink-web/pull/426

I am also cc'ing Stephan who wrote the File Source to grasp his opinion as well.

Best,

Dawid

On 17/03/2021 06:53, Chirag Dewan wrote:

Hi,

I am intending to use the File source for a production use case. I have a few use cases that are currently not supported like deleting a file once it's processed. 

So I was wondering if we can use this in production or write my own implementation? Is there any recommendations around this?

Thanks

Chirag

[Arvid Heise-4]

Hi Chirag,

we are also happy to take contributions to add features like the file deletion.

Since the file source is already running well in our deployments, I'd also recommend it to take it as a base for your custom implementation (getting a source right is not trivial).

On Thu, Mar 18, 2021 at 3:57 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

As for the issue of production readiness of the File Source(and other components) I'd recommend having a look at the PR, which is close to being merged where we express our opinion how we see certain components: https://github.com/apache/flink-web/pull/426

I am also cc'ing Stephan who wrote the File Source to grasp his opinion as well.

Best,

Dawid

On 17/03/2021 06:53, Chirag Dewan wrote:

Hi,

I am intending to use the File source for a production use case. I have a few use cases that are currently not supported like deleting a file once it's processed. 

So I was wondering if we can use this in production or write my own implementation? Is there any recommendations around this?

Thanks

Chirag