Splitting Streams

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Splitting Streams

Martin Neumann
Hej,

I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning)

The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have a lot of duplicated logic.
I'm basically looking for a single operator that can have multiple output streams of different types.

cheers Martin
Reply | Threaded
Open this post in threaded view
|

Re: Splitting Streams

Till Rohrmann
Hi Martin,

could grouping be a solution to your problem?

Cheers,
Till

On Thu, Sep 3, 2015 at 11:56 AM, Martin Neumann <[hidden email]> wrote:
Hej,

I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning)

The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have a lot of duplicated logic.
I'm basically looking for a single operator that can have multiple output streams of different types.

cheers Martin

Reply | Threaded
Open this post in threaded view
|

Re: Splitting Streams

Aljoscha Krettek
Hi Martin,

Regards,
Aljoscha

On Thu, 3 Sep 2015 at 12:02 Till Rohrmann <[hidden email]> wrote:
Hi Martin,

could grouping be a solution to your problem?

Cheers,
Till

On Thu, Sep 3, 2015 at 11:56 AM, Martin Neumann <[hidden email]> wrote:
Hej,

I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning)

The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have a lot of duplicated logic.
I'm basically looking for a single operator that can have multiple output streams of different types.

cheers Martin

Reply | Threaded
Open this post in threaded view
|

Re: Splitting Streams

Martin Neumann
@Till:
The fields are hidden inside the json string so that I have to deserialize first. Also the classes do not have so much in common. It might be possible to do it with a hierarchy of group by. I'm not sure how complicated that would be, additionally I will have to send the raw String around for longer and it might be harder to extend it later.

@Aljoscha:
The problem with the split is that the channels that is split into have to have the same type. In my case the input would be string and the output would be different types of object streams.

I think for now I will try to split the string stream into several sub-streams using the split operator and then do the type conversion afterwards. I will still have to distribute the logic over several operators this way but its less ugly as my current solution.

cheers Martin


On Thu, Sep 3, 2015 at 12:20 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi Martin,

Regards,
Aljoscha

On Thu, 3 Sep 2015 at 12:02 Till Rohrmann <[hidden email]> wrote:
Hi Martin,

could grouping be a solution to your problem?

Cheers,
Till

On Thu, Sep 3, 2015 at 11:56 AM, Martin Neumann <[hidden email]> wrote:
Hej,

I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning)

The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have a lot of duplicated logic.
I'm basically looking for a single operator that can have multiple output streams of different types.

cheers Martin