Adding Context To Logs

[Hamilton, Paul]

Hi,

 

We are trying to rework the way our Flink jobs (and really it becomes how the operators within the jobs) log.  Specifically we are trying to add contextual information to the log messages so that it is easier to filter and search for a specific jobs logs after they have been send through to our ELK stack.  I am certainly open to suggestions, and hopefully  I am missing something easier, but these were my current thoughts:

 

1)       In the application and then also within individual functions that log add the “job name” to the logging MDC in the open function so that no matter where  it is running on it can log the appropriate job name.  The problem here is that I haven’t found an easy way to get to the job name within the RichFunction.  What would be the best way to get it?

 

2)       Pass the job name (and any other contextual things we want to log) in at construction time when building the flink job to each function that needs them.  Simple, but seems kind of ugly.

 

 

Thoughts?

 

Thanks,

 

Paul Hamilton

SAP Hybris

[Kostas Kloudas]

Hello Paul,

If I understand correctly, your issues seem to be related to:

<a href="https://issues.apache.org/jira/browse/FLINK-1502?jql=assignee in (Zentol) AND text ~ &quot;metrics&quot;" class="">https://issues.apache.org/jira/browse/FLINK-1502?jql=assignee%20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

and

<a href="https://issues.apache.org/jira/browse/FLINK-2732?jql=assignee in (Zentol) AND text ~ &quot;metrics&quot;" class="">https://issues.apache.org/jira/browse/FLINK-2732?jql=assignee%20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

If this is correct, probably Chesnay (who worked on these issues) is the right person to talk to.

Kostas

On Jun 2, 2016, at 6:53 PM, Hamilton, Paul <[hidden email]> wrote:

Hi,

 

We are trying to rework the way our Flink jobs (and really it becomes how the operators within the jobs) log.  Specifically we are trying to add contextual information to the log messages so that it is easier to filter and search for a specific jobs logs after they have been send through to our ELK stack.  I am certainly open to suggestions, and hopefully  I am missing something easier, but these were my current thoughts:

 

1)       In the application and then also within individual functions that log add the “job name” to the logging MDC in the open function so that no matter where  it is running on it can log the appropriate job name.  The problem here is that I haven’t found an easy way to get to the job name within the RichFunction.  What would be the best way to get it?

 

2)       Pass the job name (and any other contextual things we want to log) in at construction time when building the flink job to each function that needs them.  Simple, but seems kind of ugly.

 

 

Thoughts?

 

Thanks,

 

Paul Hamilton

SAP Hybris

[Chesnay Schepler]

the job name is straight up not accessible from within a function. Until recently
it wasn't even accessible anywhere on the TaskManager.

To get the job name in a function you would have to pass it from the TaskManager
to the task, to the operator and (probably) to the RuntimeContext.

If you were content with using the job ID you would only have to modify the RuntimeContext.

Having these context information available in a UDF is imo a very useful thing to have.
(which TaskManager am i on, which Job/Task/Operator do i belong to)

Note that this obviously only covers the case of modifying the logging within a UDF. Anything
else would be unaffected.

Regards,
Chesnay

On 02.06.2016 22:39, Kostas Kloudas wrote:

Hello Paul,

If I understand correctly, your issues seem to be related to:

https://issues.apache.org/jira/browse/FLINK-1502?jql=assignee%20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

and

https://issues.apache.org/jira/browse/FLINK-2732?jql=assignee%20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

If this is correct, probably Chesnay (who worked on these issues) is the right person to talk to.

Kostas

On Jun 2, 2016, at 6:53 PM, Hamilton, Paul <[hidden email]> wrote:

Hi,

 

We are trying to rework the way our Flink jobs (and really it becomes how the operators within the jobs) log.  Specifically we are trying to add contextual information to the log messages so that it is easier to filter and search for a specific jobs logs after they have been send through to our ELK stack.  I am certainly open to suggestions, and hopefully  I am missing something easier, but these were my current thoughts:

 

1)       In the application and then also within individual functions that log add the “job name” to the logging MDC in the open function so that no matter where  it is running on it can log the appropriate job name.  The problem here is that I haven’t found an easy way to get to the job name within the RichFunction.  What would be the best way to get it?

 

2)       Pass the job name (and any other contextual things we want to log) in at construction time when building the flink job to each function that needs them.  Simple, but seems kind of ugly.

 

 

Thoughts?

 

Thanks,

 

Paul Hamilton

SAP Hybris