Are there pipeline API's for ETL?

[kant kodali]

Hi All,

I am wondering if there are pipeline API's for ETL?

Thanks!

[vino yang]

Hi kant,

Can you provide more context about your question? What do you mean about "pipeline API"?

IMO, you can build an ETL pipeline via composing several Flink transform APIs. About choosing which transform APIs, it depends on your business logic. 

Here are the generic APIs list.[1]

Best,

Vino

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html

kant kodali <[hidden email]> 于2020年1月11日周六 上午9:06写道：

Hi All,

I am wondering if there are pipeline API's for ETL?

Thanks!

[kant kodali]

Hi Vino,

I am new to Flink. I was thinking more like a dag builder api where I can build a dag of source,sink and transforms and hopefully fink take cares of the entire life cycle of the dag.

An example would be CDAP pipeline api.

Sent from my iPhone

On Jan 10, 2020, at 6:42 PM, vino yang <[hidden email]> wrote:



Hi kant,

Can you provide more context about your question? What do you mean about "pipeline API"?

IMO, you can build an ETL pipeline via composing several Flink transform APIs. About choosing which transform APIs, it depends on your business logic. 

Here are the generic APIs list.[1]

Best,

Vino

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html

kant kodali <[hidden email]> 于2020年1月11日周六 上午9:06写道：

Hi All,

I am wondering if there are pipeline API's for ETL?

Thanks!

[kant kodali]

Hi Vino,

Another use case would be I want to build a dag of batch sources, sinks and transforms and I want to schedule the jobs periodically. One can say similar to airflow but a Flink api would be lot better!

Sent from my iPhone

On Jan 10, 2020, at 6:42 PM, vino yang <[hidden email]> wrote:



Hi kant,

Can you provide more context about your question? What do you mean about "pipeline API"?

IMO, you can build an ETL pipeline via composing several Flink transform APIs. About choosing which transform APIs, it depends on your business logic. 

Here are the generic APIs list.[1]

Best,

Vino

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html

kant kodali <[hidden email]> 于2020年1月11日周六 上午9:06写道：

Hi All,

I am wondering if there are pipeline API's for ETL?

Thanks!