Setting app Flink logger

[Eyal Pe'er]

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[Rafi Aroch]

Hi Eyal,

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

Hope this helps,

Rafi

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[Eyal Pe'er]

Hi Rafi,

The file exists (and is the file from the official imageJ, please see below).

The user is root and it has permission. I am running in HA mode using docker.

 

cat /opt/flink/conf/log4j-console.properties

 

################################################################################

#  Licensed to the Apache Software Foundation (ASF) under one

#  or more contributor license agreements.  See the NOTICE file

#  distributed with this work for additional information

#  regarding copyright ownership.  The ASF licenses this file

#  to you under the Apache License, Version 2.0 (the

#  "License"); you may not use this file except in compliance

#  with the License.  You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

#  Unless required by applicable law or agreed to in writing, software

#  distributed under the License is distributed on an "AS IS" BASIS,

#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

#  See the License for the specific language governing permissions and

# limitations under the License.

################################################################################

 

# This affects logging for both user code and Flink

rootLogger.level = INFO

rootLogger.appenderRef.console.ref = ConsoleAppender

 

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

 

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

logger.akka.name = akka

logger.akka.level = INFO

logger.kafka.name= org.apache.kafka

logger.kafka.level = INFO

logger.hadoop.name = org.apache.hadoop

logger.hadoop.level = INFO

logger.zookeeper.name = org.apache.zookeeper

logger.zookeeper.level = INFO

 

# Log all infos to the console

appender.console.name = ConsoleAppender

appender.console.type = CONSOLE

appender.console.layout.type = PatternLayout

appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

 

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline

logger.netty.level = OFF

 

Best regards

Eyal Peer / Data Platform Developer

 

From: Rafi Aroch <[hidden email]>
Sent: Tuesday, March 10, 2020 3:55 PM
To: Eyal Pe'er <[hidden email]>
Cc: user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

Hi Eyal,

 

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

 

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

 

Hope this helps,

Rafi

 

 

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[miki haiat]

Which image are you using ?

On Tue, Mar 10, 2020, 16:27 Eyal Pe'er <[hidden email]> wrote:

Hi Rafi,

The file exists (and is the file from the official imageJ, please see below).

The user is root and it has permission. I am running in HA mode using docker.

 

cat /opt/flink/conf/log4j-console.properties

 

################################################################################

#  Licensed to the Apache Software Foundation (ASF) under one

#  or more contributor license agreements.  See the NOTICE file

#  distributed with this work for additional information

#  regarding copyright ownership.  The ASF licenses this file

#  to you under the Apache License, Version 2.0 (the

#  "License"); you may not use this file except in compliance

#  with the License.  You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

#  Unless required by applicable law or agreed to in writing, software

#  distributed under the License is distributed on an "AS IS" BASIS,

#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

#  See the License for the specific language governing permissions and

# limitations under the License.

################################################################################

 

# This affects logging for both user code and Flink

rootLogger.level = INFO

rootLogger.appenderRef.console.ref = ConsoleAppender

 

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

 

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

logger.akka.name = akka

logger.akka.level = INFO

logger.kafka.name= org.apache.kafka

logger.kafka.level = INFO

logger.hadoop.name = org.apache.hadoop

logger.hadoop.level = INFO

logger.zookeeper.name = org.apache.zookeeper

logger.zookeeper.level = INFO

 

# Log all infos to the console

appender.console.name = ConsoleAppender

appender.console.type = CONSOLE

appender.console.layout.type = PatternLayout

appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

 

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline

logger.netty.level = OFF

 

Best regards

Eyal Peer / Data Platform Developer

 

From: Rafi Aroch <[hidden email]>
Sent: Tuesday, March 10, 2020 3:55 PM
To: Eyal Pe'er <[hidden email]>
Cc: user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

Hi Eyal,

 

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

 

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

 

Hope this helps,

Rafi

 

 

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[Yang Wang]

Since you are using log4j2, the java dynamic property should not be 

"log4j.configuration". Please use "log4j.configurationFile" instead.

Maybe it is not your problem, there is something wrong with the docker

image. The log4j2 properties in "flink-console.sh" are not configured

correctly.

Best,

Yang

miki haiat <[hidden email]> 于2020年3月10日周二 下午11:50写道：

Which image are you using ?

On Tue, Mar 10, 2020, 16:27 Eyal Pe'er <[hidden email]> wrote:

Hi Rafi,

The file exists (and is the file from the official imageJ, please see below).

The user is root and it has permission. I am running in HA mode using docker.

 

cat /opt/flink/conf/log4j-console.properties

 

################################################################################

#  Licensed to the Apache Software Foundation (ASF) under one

#  or more contributor license agreements.  See the NOTICE file

#  distributed with this work for additional information

#  regarding copyright ownership.  The ASF licenses this file

#  to you under the Apache License, Version 2.0 (the

#  "License"); you may not use this file except in compliance

#  with the License.  You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

#  Unless required by applicable law or agreed to in writing, software

#  distributed under the License is distributed on an "AS IS" BASIS,

#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

#  See the License for the specific language governing permissions and

# limitations under the License.

################################################################################

 

# This affects logging for both user code and Flink

rootLogger.level = INFO

rootLogger.appenderRef.console.ref = ConsoleAppender

 

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

 

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

logger.akka.name = akka

logger.akka.level = INFO

logger.kafka.name= org.apache.kafka

logger.kafka.level = INFO

logger.hadoop.name = org.apache.hadoop

logger.hadoop.level = INFO

logger.zookeeper.name = org.apache.zookeeper

logger.zookeeper.level = INFO

 

# Log all infos to the console

appender.console.name = ConsoleAppender

appender.console.type = CONSOLE

appender.console.layout.type = PatternLayout

appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

 

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline

logger.netty.level = OFF

 

Best regards

Eyal Peer / Data Platform Developer

 

From: Rafi Aroch <[hidden email]>
Sent: Tuesday, March 10, 2020 3:55 PM
To: Eyal Pe'er <[hidden email]>
Cc: user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

Hi Eyal,

 

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

 

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

 

Hope this helps,

Rafi

 

 

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[Chesnay Schepler]

@Eyal:

The image you are using is for 1.9.2, but the logging configuration you fetched was from master.

In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead, which uses a different configuration syntax. Log4j1 pretty much ignores the entire file, causing the error.

Please use the configuration file from the release-1.9 branch.

On 11/03/2020 03:50, Yang Wang wrote:

Since you are using log4j2, the java dynamic property should not be 

"log4j.configuration". Please use "log4j.configurationFile" instead.

Maybe it is not your problem, there is something wrong with the docker

image. The log4j2 properties in "flink-console.sh" are not configured

correctly.

Best,

Yang

miki haiat <[hidden email]> 于2020年3月10日周二 下午11:50写道：

Which image are you using ?

On Tue, Mar 10, 2020, 16:27 Eyal Pe'er <[hidden email]> wrote:

Hi Rafi,

The file exists (and is the file from the official imageJ, please see below).

The user is root and it has permission. I am running in HA mode using docker.

 

cat /opt/flink/conf/log4j-console.properties

 

################################################################################

#  Licensed to the Apache Software Foundation (ASF) under one

#  or more contributor license agreements.  See the NOTICE file

#  distributed with this work for additional information

#  regarding copyright ownership.  The ASF licenses this file

#  to you under the Apache License, Version 2.0 (the

#  "License"); you may not use this file except in compliance

#  with the License.  You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

#  Unless required by applicable law or agreed to in writing, software

#  distributed under the License is distributed on an "AS IS" BASIS,

#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

#  See the License for the specific language governing permissions and

# limitations under the License.

################################################################################

 

# This affects logging for both user code and Flink

rootLogger.level = INFO

rootLogger.appenderRef.console.ref = ConsoleAppender

 

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

 

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

logger.akka.name = akka

logger.akka.level = INFO

logger.kafka.name= org.apache.kafka

logger.kafka.level = INFO

logger.hadoop.name = org.apache.hadoop

logger.hadoop.level = INFO

logger.zookeeper.name = org.apache.zookeeper

logger.zookeeper.level = INFO

 

# Log all infos to the console

appender.console.name = ConsoleAppender

appender.console.type = CONSOLE

appender.console.layout.type = PatternLayout

appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

 

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline

logger.netty.level = OFF

 

Best regards

Eyal Peer / Data Platform Developer

 

From: Rafi Aroch <[hidden email]>
Sent: Tuesday, March 10, 2020 3:55 PM
To: Eyal Pe'er <[hidden email]>
Cc: user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

Hi Eyal,

 

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

 

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

 

Hope this helps,

Rafi

 

 

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer

[Eyal Pe'er]

Thank you all.

I will change the configuration file to and deploy next week.

 

Best regards

Eyal Peer

From: Chesnay Schepler <[hidden email]>
Sent: Wednesday, March 11, 2020 11:10 AM
To: Yang Wang <[hidden email]>; miki haiat <[hidden email]>
Cc: Eyal Pe'er <[hidden email]>; Rafi Aroch <[hidden email]>; user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

@Eyal:

 

The image you are using is for 1.9.2, but the logging configuration you fetched was from master.

 

In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead, which uses a different configuration syntax. Log4j1 pretty much ignores the entire file, causing the error.

 

Please use the configuration file from the release-1.9 branch.

 

On 11/03/2020 03:50, Yang Wang wrote:

Since you are using log4j2, the java dynamic property should not be 

"log4j.configuration". Please use "log4j.configurationFile" instead.

 

Maybe it is not your problem, there is something wrong with the docker

image. The log4j2 properties in "flink-console.sh" are not configured

correctly.

 

 

Best,

Yang

 

miki haiat <[hidden email]> 于2020年3月10日周二 下午11:50写道：

Which image are you using ?

 

On Tue, Mar 10, 2020, 16:27 Eyal Pe'er <[hidden email]> wrote:

Hi Rafi,

The file exists (and is the file from the official imageJ, please see below).

The user is root and it has permission. I am running in HA mode using docker.

 

cat /opt/flink/conf/log4j-console.properties

 

################################################################################

#  Licensed to the Apache Software Foundation (ASF) under one

#  or more contributor license agreements.  See the NOTICE file

#  distributed with this work for additional information

#  regarding copyright ownership.  The ASF licenses this file

#  to you under the Apache License, Version 2.0 (the

#  "License"); you may not use this file except in compliance

#  with the License.  You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

#  Unless required by applicable law or agreed to in writing, software

#  distributed under the License is distributed on an "AS IS" BASIS,

#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

#  See the License for the specific language governing permissions and

# limitations under the License.

################################################################################

 

# This affects logging for both user code and Flink

rootLogger.level = INFO

rootLogger.appenderRef.console.ref = ConsoleAppender

 

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

 

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

logger.akka.name = akka

logger.akka.level = INFO

logger.kafka.name= org.apache.kafka

logger.kafka.level = INFO

logger.hadoop.name = org.apache.hadoop

logger.hadoop.level = INFO

logger.zookeeper.name = org.apache.zookeeper

logger.zookeeper.level = INFO

 

# Log all infos to the console

appender.console.name = ConsoleAppender

appender.console.type = CONSOLE

appender.console.layout.type = PatternLayout

appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

 

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline

logger.netty.level = OFF

 

Best regards

Eyal Peer / Data Platform Developer

 

From: Rafi Aroch <[hidden email]>
Sent: Tuesday, March 10, 2020 3:55 PM
To: Eyal Pe'er <[hidden email]>
Cc: user <[hidden email]>; StartApp R&D Data Platform <[hidden email]>
Subject: Re: Setting app Flink logger

 

Hi Eyal,

 

Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file.

You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids.

 

If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup.

 

Hope this helps,

Rafi

 

 

On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[hidden email]> wrote:

Hi,

I am running Flink in YARN mode using the official image with few additional files.

I’ve noticed that my logger failed to initialize:

 

root:~# docker logs flink-task-manager

Starting taskexecutor as a console application on host ***.

log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

I followed the documentation and seems like all related configuration files exist.

Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf

 

In addition, seems like the process got the right parameters:

root     21892 21866  1 08:29 ?        00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

 

What am I doing wrong ? How can I turn logger on?

Best regards

Eyal Peer