Flink 1.11 not showing logs

[Diwakar Jha]

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Yang Wang]

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Diwakar Jha]

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Yang Wang]

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Diwakar Jha]

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Yang Wang]

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Diwakar Jha]

HI Yang,

I'm able to see taskmanage and jobmanager logs after I changed the log4j.properties file (/usr/lib/flink/conf). Thank you!

 I updated the file as shown below. I had to kill the app ( yarn application -kill <appid> ) and start flink job again to get the logs. This doesn't seem like an efficient way. I was wondering if there's a more simpler way to do it in production. let me know, please!

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

On Tue, Nov 3, 2020 at 4:56 AM Yang Wang <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[Yang Wang]

If you have set the environment FLINK_CONF_DIR, then it will have a higher priority.

I think that could be why you changed the log4j.properties in the conf directory but it does

not take effect.

Yes, if you have changed the log4j.properties, you need to relaunch the Flink application.

Although we have a ticket to support updating the log level via rest API[1], we do not have

much process on this. And I think you could keep an eye on this and share your thoughts.

[1]. https://issues.apache.org/jira/browse/FLINK-16478

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月12日周四 上午5:38写道：

HI Yang,

I'm able to see taskmanage and jobmanager logs after I changed the log4j.properties file (/usr/lib/flink/conf). Thank you!

 I updated the file as shown below. I had to kill the app ( yarn application -kill <appid> ) and start flink job again to get the logs. This doesn't seem like an efficient way. I was wondering if there's a more simpler way to do it in production. let me know, please!

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

On Tue, Nov 3, 2020 at 4:56 AM Yang Wang <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

[马阳阳]

Hi Yang,

We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs.

I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set.

When checking the java dynamic options of task manager, I found that the log related options are not

set.

This is the output when ussuing "ps -ef | grep <container_id>".

yarn     31049 30974  9 13:57 ?        00:03:31 /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 org.apache.flink.yarn.YarnTaskExecutorRunner -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=134217730b -D taskmanager.memory.network.min=134217730b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D taskmanager.memory.task.heap.size=402653174b -D taskmanager.memory.task.off-heap.size=0b --configDir . -Djobmanager.rpc.address=dhpdn09-113 -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0 -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63 -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113

My question is, what maybe the problem for this? And any suggestions?

By the way, we submit the program from Java program instead of from the command line.

Thanks.

ps: I sent the mail to spark user mail list un-attentionally. So I resent it to the Flink user mail list. Sorry for the inconvenience to  @Yang Wang 

At 2020-11-03 20:56:19, "Yang Wang" <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

 

[Yang Wang]

If you are using your own deployer(aka a java program calls the Flink client API to submit Flink jobs),

you need to check the jobmanager configuration in webUI whether "$internal.yarn.log-config-file" 

is correctly set. If not, maybe you need to set "$internal.deployment.config-dir" in your deployer,

not simply set the FLINK_CONF_DIR environment. Because your deployer needs to do some configuration setting

which CliFrontend has done. Please have a try and share more feedback.

Best,

Yang

马阳阳 <[hidden email]> 于2020年11月16日周一 下午2:47写道：

Hi Yang,

We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs.

I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set.

When checking the java dynamic options of task manager, I found that the log related options are not

set.

This is the output when ussuing "ps -ef | grep <container_id>".

yarn     31049 30974  9 13:57 ?        00:03:31 /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 org.apache.flink.yarn.YarnTaskExecutorRunner -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=134217730b -D taskmanager.memory.network.min=134217730b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D taskmanager.memory.task.heap.size=402653174b -D taskmanager.memory.task.off-heap.size=0b --configDir . -Djobmanager.rpc.address=dhpdn09-113 -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0 -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63 -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113

My question is, what maybe the problem for this? And any suggestions?

By the way, we submit the program from Java program instead of from the command line.

Thanks.

ps: I sent the mail to spark user mail list un-attentionally. So I resent it to the Flink user mail list. Sorry for the inconvenience to  @Yang Wang 

At 2020-11-03 20:56:19, "Yang Wang" <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

 

[LINZ, Arnaud]

Hello,

I'm running Flink 1.10 on a yarn cluster. I have a streaming application, that, when under heavy load, fails from time to time with this unique error message in the whole yarn log:

(...)
2020-11-15 16:18:42,202 WARN  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Received late message for now expired checkpoint attempt 63 from task 4cbc940112a596db54568b24f9209aac of job 1e1717d19bd8ea296314077e42e1c7e5 at container_e38_1604477334666_0960_01_000004 @ xxx (dataPort=33099).
2020-11-15 16:18:55,043 INFO  org.apache.flink.yarn.YarnResourceManager                     - Closing TaskExecutor connection container_e38_1604477334666_0960_01_000004 because: The TaskExecutor is shutting down.
2020-11-15 16:18:55,087 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Map (7/15) (c8e92cacddcd4e41f51a2433d07d2153) switched from RUNNING to FAILED.
org.apache.flink.util.FlinkException: The TaskExecutor is shutting down.

      at org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:359)
        at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:218)
        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:509)
        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:175)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
        at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
        at akka.actor.ActorCell.invoke(ActorCell.scala:561)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
        at akka.dispatch.Mailbox.run(Mailbox.scala:225)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-11-15 16:18:55,092 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - Calculating tasks to restart to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6.
2020-11-15 16:18:55,101 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - 230 tasks should be restarted to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6.
(...)

What could be the cause of this failure? Why is there no other error message?

I've tried to increase the value of heartbeat.timeout, thinking that maybe it was due to a slow responding mapper, but it did not solve the issue.

Best regards,
Arnaud

________________________________

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[马阳阳]

Hi Yang,

Thank you for you reply.

I set the value for "$internal.deployment.config-dir" to the Flink configuration directory.

And the configuration showed on Flink web UI. But it still not work. So I wonder what should 

I set as the value for "$internal.deployment.config-dir"?

At 2020-11-16 16:43:11, "Yang Wang" <[hidden email]> wrote:

If you are using your own deployer(aka a java program calls the Flink client API to submit Flink jobs),

you need to check the jobmanager configuration in webUI whether "$internal.yarn.log-config-file" 

is correctly set. If not, maybe you need to set "$internal.deployment.config-dir" in your deployer,

not simply set the FLINK_CONF_DIR environment. Because your deployer needs to do some configuration setting

which CliFrontend has done. Please have a try and share more feedback.

Best,

Yang

马阳阳 <[hidden email]> 于2020年11月16日周一 下午2:47写道：

Hi Yang,

We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs.

I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set.

When checking the java dynamic options of task manager, I found that the log related options are not

set.

This is the output when ussuing "ps -ef | grep <container_id>".

yarn     31049 30974  9 13:57 ?        00:03:31 /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 org.apache.flink.yarn.YarnTaskExecutorRunner -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=134217730b -D taskmanager.memory.network.min=134217730b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D taskmanager.memory.task.heap.size=402653174b -D taskmanager.memory.task.off-heap.size=0b --configDir . -Djobmanager.rpc.address=dhpdn09-113 -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0 -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63 -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113

My question is, what maybe the problem for this? And any suggestions?

By the way, we submit the program from Java program instead of from the command line.

Thanks.

ps: I sent the mail to spark user mail list un-attentionally. So I resent it to the Flink user mail list. Sorry for the inconvenience to  @Yang Wang 

At 2020-11-03 20:56:19, "Yang Wang" <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

 

 

[马阳阳]

Hi Yang,

I checked the `YarnLogConfigUtil`, it does some work to set the configuration for log.

Should I copy the logic to my deployer?

At 2020-11-16 17:21:07, "马阳阳" <[hidden email]> wrote:

Hi Yang,

Thank you for you reply.

I set the value for "$internal.deployment.config-dir" to the Flink configuration directory.

And the configuration showed on Flink web UI. But it still not work. So I wonder what should 

I set as the value for "$internal.deployment.config-dir"?

At 2020-11-16 16:43:11, "Yang Wang" <[hidden email]> wrote:

If you are using your own deployer(aka a java program calls the Flink client API to submit Flink jobs),

you need to check the jobmanager configuration in webUI whether "$internal.yarn.log-config-file" 

is correctly set. If not, maybe you need to set "$internal.deployment.config-dir" in your deployer,

not simply set the FLINK_CONF_DIR environment. Because your deployer needs to do some configuration setting

which CliFrontend has done. Please have a try and share more feedback.

Best,

Yang

马阳阳 <[hidden email]> 于2020年11月16日周一 下午2:47写道：

Hi Yang,

We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs.

I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set.

When checking the java dynamic options of task manager, I found that the log related options are not

set.

This is the output when ussuing "ps -ef | grep <container_id>".

yarn     31049 30974  9 13:57 ?        00:03:31 /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 org.apache.flink.yarn.YarnTaskExecutorRunner -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=134217730b -D taskmanager.memory.network.min=134217730b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D taskmanager.memory.task.heap.size=402653174b -D taskmanager.memory.task.off-heap.size=0b --configDir . -Djobmanager.rpc.address=dhpdn09-113 -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0 -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63 -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113

My question is, what maybe the problem for this? And any suggestions?

By the way, we submit the program from Java program instead of from the command line.

Thanks.

ps: I sent the mail to spark user mail list un-attentionally. So I resent it to the Flink user mail list. Sorry for the inconvenience to  @Yang Wang 

At 2020-11-03 20:56:19, "Yang Wang" <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

 

 

 

[马阳阳]

Hi Yang,

After I copied the logic from `YarnLogConfigUtil` to my own deployer (maybe call its logic instead of copying is a better option), the logs now can show normally.

Thanks again for the kind help.

At 2020-11-16 17:28:47, "马阳阳" <[hidden email]> wrote:

Hi Yang,

I checked the `YarnLogConfigUtil`, it does some work to set the configuration for log.

Should I copy the logic to my deployer?

At 2020-11-16 17:21:07, "马阳阳" <[hidden email]> wrote:

Hi Yang,

Thank you for you reply.

I set the value for "$internal.deployment.config-dir" to the Flink configuration directory.

And the configuration showed on Flink web UI. But it still not work. So I wonder what should 

I set as the value for "$internal.deployment.config-dir"?

At 2020-11-16 16:43:11, "Yang Wang" <[hidden email]> wrote:

If you are using your own deployer(aka a java program calls the Flink client API to submit Flink jobs),

you need to check the jobmanager configuration in webUI whether "$internal.yarn.log-config-file" 

is correctly set. If not, maybe you need to set "$internal.deployment.config-dir" in your deployer,

not simply set the FLINK_CONF_DIR environment. Because your deployer needs to do some configuration setting

which CliFrontend has done. Please have a try and share more feedback.

Best,

Yang

马阳阳 <[hidden email]> 于2020年11月16日周一 下午2:47写道：

Hi Yang,

We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs.

I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set.

When checking the java dynamic options of task manager, I found that the log related options are not

set.

This is the output when ussuing "ps -ef | grep <container_id>".

yarn     31049 30974  9 13:57 ?        00:03:31 /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902 -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456 org.apache.flink.yarn.YarnTaskExecutorRunner -D taskmanager.memory.framework.off-heap.size=134217728b -D taskmanager.memory.network.max=134217730b -D taskmanager.memory.network.min=134217730b -D taskmanager.memory.framework.heap.size=134217728b -D taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D taskmanager.memory.task.heap.size=402653174b -D taskmanager.memory.task.off-heap.size=0b --configDir . -Djobmanager.rpc.address=dhpdn09-113 -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0 -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63 -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113

My question is, what maybe the problem for this? And any suggestions?

By the way, we submit the program from Java program instead of from the command line.

Thanks.

ps: I sent the mail to spark user mail list un-attentionally. So I resent it to the Flink user mail list. Sorry for the inconvenience to  @Yang Wang 

At 2020-11-03 20:56:19, "Yang Wang" <[hidden email]> wrote:

You could issue "ps -ef | grep container_id_for_some_tm". And then you will find the

following java options about log4j.

-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log

-Dlog4j.configuration=file:./log4j.properties

-Dlog4j.configurationFile=file:./log4j.properties

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午11:37写道：

Sure. I will check that and get back to you. could you please share how to check java dynamic options?

Best,

Diwakar

On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <[hidden email]> wrote:

If you have already updated the log4j.properties, and it still could not work, then I

suggest to log in the Yarn NodeManager machine and check the log4j.properties

in the container workdir is correct. Also you could have a look at the java dynamic

options are correctly set.

I think it should work if the log4j.properties and java dynamic options are set correctly.

BTW, could you share the new yarn logs?

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 下午4:32写道：

Hi Yang,

Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs.

Actual
log4j.rootLogger=INFO,file

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

modified : commented the above and added new logging from actual flink application log4.properties file

#log4j.rootLogger=INFO,file

# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF

**********************************

I also think its related to the log4j setting but I'm not able to figure it out. 

Please let me know if you want any other log files or configuration. 

Thanks.

On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <[hidden email]> wrote:

Hi Diwakar Jha,

From the logs you have provided, everything seems working as expected. The JobManager and TaskManager

java processes have been started with correct dynamic options, especially for the logging.

Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the

log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11.

Best,

Yang

Diwakar Jha <[hidden email]> 于2020年11月2日周一 上午1:57写道：

Hi

I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. 

I see the below error in stdout.

18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1604033334508_0001_01_000004.

java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG does not exist on the TaskExecutor.

I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me.

i have the following setup : 

a) i'm deploying flink using yarn. I have attached yarn application id logs.

c) stsd setup

metrics.reporters: stsd
metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

 

 

 

 

[Yang Wang]

Hi Arnaud,

It seems that the TaskExecutor terminated exceptionally. I think you need to check the logs of

container_e38_1604477334666_0960_01_000004 to figure out why it crashed or shut down.

Best,

Yang

LINZ, Arnaud <[hidden email]> 于2020年11月16日周一 下午7:11写道：

Hello,

I'm running Flink 1.10 on a yarn cluster. I have a streaming application, that, when under heavy load, fails from time to time with this unique error message in the whole yarn log:

(...)
2020-11-15 16:18:42,202 WARN  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Received late message for now expired checkpoint attempt 63 from task 4cbc940112a596db54568b24f9209aac of job 1e1717d19bd8ea296314077e42e1c7e5 at container_e38_1604477334666_0960_01_000004 @ xxx (dataPort=33099).
2020-11-15 16:18:55,043 INFO  org.apache.flink.yarn.YarnResourceManager                     - Closing TaskExecutor connection container_e38_1604477334666_0960_01_000004 because: The TaskExecutor is shutting down.
2020-11-15 16:18:55,087 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Map (7/15) (c8e92cacddcd4e41f51a2433d07d2153) switched from RUNNING to FAILED.
org.apache.flink.util.FlinkException: The TaskExecutor is shutting down.

      at org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:359)
        at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:218)
        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:509)
        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:175)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
        at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
        at akka.actor.ActorCell.invoke(ActorCell.scala:561)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
        at akka.dispatch.Mailbox.run(Mailbox.scala:225)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-11-15 16:18:55,092 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - Calculating tasks to restart to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6.
2020-11-15 16:18:55,101 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy  - 230 tasks should be restarted to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6.
(...)

What could be the cause of this failure? Why is there no other error message?

I've tried to increase the value of heartbeat.timeout, thinking that maybe it was due to a slow responding mapper, but it did not solve the issue.

Best regards,
Arnaud

________________________________

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.