Further aggregation possible after sink?

[Mohit Anchlia]

Is it possible to further add aggregation after the sink task executes? Or is the sink the last stage of the workflow? Is this flow possible?

start stream -> transform -> load (sink) -> mark final state as loaded in a table after all the load was successful in previous state (sink)

[abhishekrs]

You can keep adding stages, but then your sink is no more a sink - it would have transformed into a map or a flatmap !

On Mon, Feb 13, 2017 at 12:34 PM Mohit Anchlia <[hidden email]> wrote:

Is it possible to further add aggregation after the sink task executes? Or is the sink the last stage of the workflow? Is this flow possible?

start stream -> transform -> load (sink) -> mark final state as loaded in a table after all the load was successful in previous state (sink)

[Mohit Anchlia]

I am assuming it's a bad idea to have DB operations in a map? May be a better way is to chain the pipelines. Is that possible? eg:

workflow A ends -> workflow B starts

On Mon, Feb 13, 2017 at 12:38 PM, Abhishek Singh <[hidden email]> wrote:

You can keep adding stages, but then your sink is no more a sink - it would have transformed into a map or a flatmap !

On Mon, Feb 13, 2017 at 12:34 PM Mohit Anchlia <[hidden email]> wrote:

Is it possible to further add aggregation after the sink task executes? Or is the sink the last stage of the workflow? Is this flow possible?

start stream -> transform -> load (sink) -> mark final state as loaded in a table after all the load was successful in previous state (sink)