Flink Stream assign unique transaction ID

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Stream assign unique transaction ID

tanguy racinet
Hi,

I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions to track changes and mine Frequent Item Sets.

I identified several potential solutions such as:
 - Check pointing Local variables (snapshotState and restoreState for once only passage)
 - Key Value State Interface (using operatorState and getRuntimeContext but is deprecated)
 - broadcast function
 - Co-Map function

After trying all of them, the only one I could actually make work for my needs uses the deprecated operatorState.
I tried using co-map function but couldn’t the number from one map and use it in the other mapper to modify the Tid value.

Is there any solution I haven’t thought of in order to assign a unique Long to every one of my transactions within the DataStream and to ensure they are not only unique but also ordered and linearly distributed (transaction1 -> id = 1; transaction2 -> id = 2; etc)
Or maybe some working examples of the solutions I tried but couldn’t obtain what I needed with it ?

Thank you in advance.
Regards,
Tanguy R.
Reply | Threaded
Open this post in threaded view
|

Re: Flink Stream assign unique transaction ID

Aljoscha Krettek
Hi,
OperatorState was discarded in favor of ValueState, which you can retrieve using RuntimeContext.getState(…). This provides the same functionality as OperatorState but is the more future proof interface.

Cheers,
Aljoscha

> On 19 Feb 2016, at 15:38, Tanguy Racinet <[hidden email]> wrote:
>
> Hi,
>
> I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions to track changes and mine Frequent Item Sets.
>
> I identified several potential solutions such as:
> - Check pointing Local variables (snapshotState and restoreState for once only passage)
> - Key Value State Interface (using operatorState and getRuntimeContext but is deprecated)
> - broadcast function
> - Co-Map function
>
> After trying all of them, the only one I could actually make work for my needs uses the deprecated operatorState.
> I tried using co-map function but couldn’t the number from one map and use it in the other mapper to modify the Tid value.
>
> Is there any solution I haven’t thought of in order to assign a unique Long to every one of my transactions within the DataStream and to ensure they are not only unique but also ordered and linearly distributed (transaction1 -> id = 1; transaction2 -> id = 2; etc)
> Or maybe some working examples of the solutions I tried but couldn’t obtain what I needed with it ?
>
> Thank you in advance.
> Regards,
> Tanguy R.