Log4j configuration on YARN

[Nick Dimiduk]

Can anyone tell me where I must place my application-specific log4j.properties to have them honored when running on a YARN cluster? In my application jar doesn't work. In the log4j files under flink/conf doesn't work.

My goal is to set the log level for 'com.mycompany' classes used in my flink application to DEBUG.

Thanks,

Nick

[Ufuk Celebi]

Hey Nick!

I just checked and the conf/log4j.properties file is copied and is
given as an argument to the JVM.

You should see the following:
- client logs that the conf/log4j.properties file is copied
- JobManager logs show log4j.configuration being passed to the JVM.

Can you confirm that these shows up? If yes, but you still don't get
the expected logging, I would check via -Dlog4j.debug what is
configured (prints to stdout I think). Does this help?

– Ufuk


On Fri, Mar 11, 2016 at 6:02 PM, Nick Dimiduk <[hidden email]> wrote:

> Can anyone tell me where I must place my application-specific
> log4j.properties to have them honored when running on a YARN cluster? In my
> application jar doesn't work. In the log4j files under flink/conf doesn't
> work.
>
> My goal is to set the log level for 'com.mycompany' classes used in my flink
> application to DEBUG.
>
> Thanks,
> Nick
>

[rmetzger0]

Hi Nick,

the name of the "log4j-yarn-session.properties" file might be a bit misleading. The file is just used for the YARN session client, running locally.

The Job- and TaskManager are going to use the log4j.properties on the cluster.

On Fri, Mar 11, 2016 at 7:20 PM, Ufuk Celebi <[hidden email]> wrote:

Hey Nick!

I just checked and the conf/log4j.properties file is copied and is
given as an argument to the JVM.

You should see the following:
- client logs that the conf/log4j.properties file is copied
- JobManager logs show log4j.configuration being passed to the JVM.

Can you confirm that these shows up? If yes, but you still don't get
the expected logging, I would check via -Dlog4j.debug what is
configured (prints to stdout I think). Does this help?

– Ufuk

On Fri, Mar 11, 2016 at 6:02 PM, Nick Dimiduk <[hidden email]> wrote:
> Can anyone tell me where I must place my application-specific
> log4j.properties to have them honored when running on a YARN cluster? In my
> application jar doesn't work. In the log4j files under flink/conf doesn't
> work.
>
> My goal is to set the log level for 'com.mycompany' classes used in my flink
> application to DEBUG.
>
> Thanks,
> Nick
>