Flink Logging on EMR

[mars]

Hi,

  I am running my Flink jobs on EMR and i didn't include any
log4j.properties as part of my JAR and i am using slf4j (and included the
dependent jars in the uber jar i created) and logging in my app.

  When i run my everything is running fine except i cannot find my
application logs any where

 I am running the Flink job/app with (-p 2) i see two task managers and when
i looked into the logs (none of the app specific logs can be found in those
logs). We are using the INFO Level logging.

I was hoping the logs will go to default Console Appender.

In the Master node Flink Conf i have found logback-console.xml (which sets
the root level logging to INFO) and is using Console Appender and there is
also log4j.properties file which also sets the Root Level logging to INFO
and is using FileAppender

I also tried to access the logs using "yarn logs --applicationId <>" i am
getting

$ yarn logs -applicationId application_1593579475717_0001
20/07/01 21:16:32 INFO client.RMProxy: Connecting to ResourceManager at
<>:8032
/var/log/hadoop-yarn/apps/root/logs/application_1593579475717_0001 does not
exist.
Log aggregation has not completed or is not enabled.

And Yarn Log Aggregation is already enabled. When i checked
/etc/hadoop/conf/yarn-site.xml

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

It might be the case that i can only see the logs through yarn once the
application completes/finishes/fails

Thanks
Sateesh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Ken Krugler]

Hi Sateesh,

Note that there are three classes of log files, when running Flink on EMR:

1. The output from the main class.

Since I typically run the job by sshing onto the master and using the CLI from there, I have control over where that output goes. E.g.

nohup bin/flink run -m yarn-cluster -yn 48 /path/to/my-job.jar >> my.log 2>&1 &

And then:

tail -f my.log

2. Logging by the JobManager

The JobManager log is available via the Yarn Application Overview screen (see the Logs link in the attempt list near the bottom). When your tool fails (e.g., due to a missing command-line argument), the error output is available via the stderr link in that Step of the EMR Cluster > Steps tab

3. Logging by each TaskManager

I typically log into the slave to have a look at the Task Manager error/status output (e.g., in /var/log/hadoop-yarn/containers/application_1546471484145_0002/container_1546471484145_0002_01_000002/taskmanager.err). One common approach here is to grep the taskmanager.log files (on each slave), e.g.

sudo find /var/log/hadoop-yarn/containers/application_1568579660214_0004/ -name "taskmanager.log" | sudo xargs grep “text of interest”

HTH,

— Ken

 

On Jul 2, 2020, at 9:29 AM, mars <[hidden email]> wrote:

Hi,

 I am running my Flink jobs on EMR and i didn't include any
log4j.properties as part of my JAR and i am using slf4j (and included the
dependent jars in the uber jar i created) and logging in my app.

 When i run my everything is running fine except i cannot find my
application logs any where

I am running the Flink job/app with (-p 2) i see two task managers and when
i looked into the logs (none of the app specific logs can be found in those
logs). We are using the INFO Level logging.

I was hoping the logs will go to default Console Appender.

In the Master node Flink Conf i have found logback-console.xml (which sets
the root level logging to INFO) and is using Console Appender and there is
also log4j.properties file which also sets the Root Level logging to INFO
and is using FileAppender

I also tried to access the logs using "yarn logs --applicationId <>" i am
getting

$ yarn logs -applicationId application_1593579475717_0001
20/07/01 21:16:32 INFO client.RMProxy: Connecting to ResourceManager at
<>:8032
/var/log/hadoop-yarn/apps/root/logs/application_1593579475717_0001 does not
exist.
Log aggregation has not completed or is not enabled.

And Yarn Log Aggregation is already enabled. When i checked
/etc/hadoop/conf/yarn-site.xml

<property>
   <name>yarn.log-aggregation-enable</name>
   <value>true</value>
</property>

It might be the case that i can only see the logs through yarn once the
application completes/finishes/fails

Thanks
Sateesh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--------------------------

Ken Krugler

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[KristoffSC]

Hi Mars,
Were you able to solve this problem?

I'm facing exact same issue. I dont see logs from taskmanager from my
operators (taskmnager.out file) on EMR although running this locally from
IDE logs are printed.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Arvid Heise-3]

Hi KristoffSC,

taskmanager.out should only show the output of the process starting the taskmanager. In most cases, you probably want to look into taskmanager.log.

On Tue, Dec 29, 2020 at 3:42 PM KristoffSC <[hidden email]> wrote:

Hi Mars,
Were you able to solve this problem?

I'm facing exact same issue. I dont see logs from taskmanager from my
operators (taskmnager.out file) on EMR although running this locally from
IDE logs are printed.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

--