Trying to detecting changes

[toletum]

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

[Aljoscha Krettek]

Hi,

this could be done by implementing a user function that keeps state or by using windows with a custom Trigger. On only works, however, if you only have one Kafka partition and if your Flink job is executing with parallelism=1. Otherwise we don't have any ordering guarantees on streams.

Cheers,

Aljoscha

On Wed, 20 Apr 2016 at 12:50 <[hidden email]> wrote:

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

[stefanobaghino]

Can the CEP library be used for this use case?

On Wed, Apr 20, 2016 at 2:02 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

this could be done by implementing a user function that keeps state or by using windows with a custom Trigger. On only works, however, if you only have one Kafka partition and if your Flink job is executing with parallelism=1. Otherwise we don't have any ordering guarantees on streams.

Cheers,

Aljoscha

On Wed, 20 Apr 2016 at 12:50 <[hidden email]> wrote:

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

--

[Till Rohrmann]

You could use CEP for that. First you would create a pattern of two states which matches everything. In the select function you could then check whether both elements are different. 

However, this would be a little bit of an overkill for this simple use case. You could for example simply use a flat map operation which stores the last seen element. Then whenever you see a different element you can emit a change event.

Cheers,

Till

On Wed, Apr 20, 2016 at 2:43 PM, Stefano Baghino <[hidden email]> wrote:

Can the CEP library be used for this use case?

On Wed, Apr 20, 2016 at 2:02 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

this could be done by implementing a user function that keeps state or by using windows with a custom Trigger. On only works, however, if you only have one Kafka partition and if your Flink job is executing with parallelism=1. Otherwise we don't have any ordering guarantees on streams.

Cheers,

Aljoscha

On Wed, 20 Apr 2016 at 12:50 <[hidden email]> wrote:

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

--

BR,

Stefano Baghino

Software Engineer @ Radicalbit

[stefanobaghino]

Ok, thanks for the clarification Till.

On Wed, Apr 20, 2016 at 4:46 PM, Till Rohrmann <[hidden email]> wrote:

You could use CEP for that. First you would create a pattern of two states which matches everything. In the select function you could then check whether both elements are different. 

However, this would be a little bit of an overkill for this simple use case. You could for example simply use a flat map operation which stores the last seen element. Then whenever you see a different element you can emit a change event.

Cheers,

Till

On Wed, Apr 20, 2016 at 2:43 PM, Stefano Baghino <[hidden email]> wrote:

Can the CEP library be used for this use case?

On Wed, Apr 20, 2016 at 2:02 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

this could be done by implementing a user function that keeps state or by using windows with a custom Trigger. On only works, however, if you only have one Kafka partition and if your Flink job is executing with parallelism=1. Otherwise we don't have any ordering guarantees on streams.

Cheers,

Aljoscha

On Wed, 20 Apr 2016 at 12:50 <[hidden email]> wrote:

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

--

BR,

Stefano Baghino

Software Engineer @ Radicalbit

--

[toletum]

Thanks Till,

At the end, I'm going to use a countWindowAll(2,1) and RichAllWindowFunction.

Regards,

On mié., abr. 20, 2016 at 16:46, Till Rohrmann <[hidden email]> wrote:

You could use CEP for that. First you would create a pattern of two states which matches everything. In the select function you could then check whether both elements are different. 

However, this would be a little bit of an overkill for this simple use case. You could for example simply use a flat map operation which stores the last seen element. Then whenever you see a different element you can emit a change event.

Cheers,

Till

On Wed, Apr 20, 2016 at 2:43 PM, Stefano Baghino <[hidden email]> wrote:

Can the CEP library be used for this use case?

On Wed, Apr 20, 2016 at 2:02 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

this could be done by implementing a user function that keeps state or by using windows with a custom Trigger. On only works, however, if you only have one Kafka partition and if your Flink job is executing with parallelism=1. Otherwise we don't have any ordering guarantees on streams.

Cheers,

Aljoscha

On Wed, 20 Apr 2016 at 12:50 <[hidden email]> wrote:

Hi! I'm a beginner in Flink.
I'm reading from a Kafka topic. In this topic, I receive a character each event, like that:

Event.: 1 2 3 4 5 6 7 8 9...
Data..: A A A B B B B C C...

I would like to do a "trigger" when the character is different than before. For example:
Event º1 fire because of A is different to "null"
Event º4 fire because of B is different to A
Event º8 fire because of C is different to B

Could it be possible?

--

BR,

Stefano Baghino

Software Engineer @ Radicalbit