How to interact with a running flink application?

[Anchit Jatana]

Hi All,

Brief: I have a use case where I need to interact with a running flink application.

Detail:

My Flink application has a Kafka source, an operator processing on the content received from the Kafka stream(this operator is using a lookup from an external source file to accomplish the processing of the Kafka content). If the content of the file kept at the same source location changes, I need to notify the operator to update its lookup content loaded in the memory and continue its processing of Kafka content with the new loaded lookup content without stopping the Flink application.

Is there a way where I can "interact with the running Flink Application" through some event or something to notify the application to make some changes in its operation without stopping the application.

Thank you!

Regards,

Anchit

[Ufuk Celebi]

Hey Anchit,

the usual recommendation for this is to use a CoMap/CoFlatMap
operator, where the second input are the lookup location changes. You
can then use this input to update the location.

Search for CoMap/CoFlatMap here:
https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#datastream-transformations

Best,

Ufuk

On Wed, Sep 28, 2016 at 9:37 AM, Anchit Jatana
<[hidden email]> wrote:

> Hi All,
>
> Brief: I have a use case where I need to interact with a running flink
> application.
>
> Detail:
>
> My Flink application has a Kafka source, an operator processing on the
> content received from the Kafka stream(this operator is using a lookup from
> an external source file to accomplish the processing of the Kafka content).
> If the content of the file kept at the same source location changes, I need
> to notify the operator to update its lookup content loaded in the memory and
> continue its processing of Kafka content with the new loaded lookup content
> without stopping the Flink application.
>
> Is there a way where I can "interact with the running Flink Application"
> through some event or something to notify the application to make some
> changes in its operation without stopping the application.
>
> Thank you!
>
> Regards,
> Anchit

[Anchit Jatana]

Hi Ufuk,

Thanks for your help, I'm working on using the suggested approach to address my use case.

Regards,

Anchit

On Wed, Sep 28, 2016 at 12:48 AM, Ufuk Celebi <[hidden email]> wrote:

Hey Anchit,

the usual recommendation for this is to use a CoMap/CoFlatMap
operator, where the second input are the lookup location changes. You
can then use this input to update the location.

Search for CoMap/CoFlatMap here:
https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#datastream-transformations

Best,

Ufuk

On Wed, Sep 28, 2016 at 9:37 AM, Anchit Jatana
<[hidden email]> wrote:
> Hi All,
>
> Brief: I have a use case where I need to interact with a running flink
> application.
>
> Detail:
>
> My Flink application has a Kafka source, an operator processing on the
> content received from the Kafka stream(this operator is using a lookup from
> an external source file to accomplish the processing of the Kafka content).
> If the content of the file kept at the same source location changes, I need
> to notify the operator to update its lookup content loaded in the memory and
> continue its processing of Kafka content with the new loaded lookup content
> without stopping the Flink application.
>
> Is there a way where I can "interact with the running Flink Application"
> through some event or something to notify the application to make some
> changes in its operation without stopping the application.
>
> Thank you!
>
> Regards,
> Anchit