Flink app logs to Elastic Search

[bat man]

Hi Team,

I have a requirement to push the flink app logs to Elastic Search for log management. Can anyone guide if you are already doing this.

I have tried this -

https://cristian.io/post/flink-log4j/

I’m not getting any error for a sample job I tried.

I am using EMR to run Flink 1.9 and Elastic Search latest version running on ec2 machine.

Thanks,

Hemant

[Aljoscha Krettek]

So you're saying there is no logging output whatsoever being sent to
Elasticsearch? Did you try and see if the jar file is being picked up?  
Are you still getting the pre-defined, text-based logging output?

Best,
Aljoscha

On 2021/01/07 17:04, bat man wrote:

>Hi Team,
>
>I have a requirement to push the flink app logs to Elastic Search for log
>management. Can anyone guide if you are already doing this.
>I have tried this -
>https://cristian.io/post/flink-log4j/
>I’m not getting any error for a sample job I tried.
>
>I am using EMR to run Flink 1.9 and Elastic Search latest version running
>on ec2 machine.
>
>Thanks,
>Hemant

[Aljoscha Krettek]

On 2021/01/11 01:29, bat man wrote:
>Yes, no entries to the elastic search. No indices were created in elastic.
>Jar is getting picked up which I can see from yarn logs. Pre-defined text
>based logging is also available.

Hmm, I can't imagine much that could go wrong. Maybe there is some
interference from other configuration files. Could you try and make sure
that you only have the configuration and logging system in the classpath
that you want to use?

Best,
Aljoscha

[bat man]

I was able to make it work with a fresh Elastic installation. Now taskmanager and jobmanager logs are available in elastic.

Thanks for the pointers.

-Hemant. 

On Wed, Jan 13, 2021 at 6:21 PM Aljoscha Krettek <[hidden email]> wrote:

On 2021/01/11 01:29, bat man wrote:
>Yes, no entries to the elastic search. No indices were created in elastic.
>Jar is getting picked up which I can see from yarn logs. Pre-defined text
>based logging is also available.

Hmm, I can't imagine much that could go wrong. Maybe there is some
interference from other configuration files. Could you try and make sure
that you only have the configuration and logging system in the classpath
that you want to use?

Best,
Aljoscha

[Aljoscha Krettek]

On 2021/01/15 10:43, bat man wrote:
>I was able to make it work with a fresh Elastic installation. Now
>taskmanager and jobmanager logs are available in elastic.
>Thanks for the pointers.

Thanks for letting us know!