Where does the logs in Flink GUI's Exception tab come from?

[Averell]

Hi everyone,

I am running Flink in EMR YARN cluster, and when the job failed and
restarted, I could see some logs in the Exception tab of Flink GUI.
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/Screen_Shot_2019-03-15_at_12.png>

I could not find this piece of logs on my cluster's hard-disk - not in TM or
JM logs.

Where can I find this?

Thanks.

Here below is my logback.xml. I'm not sure it has anything to do with my
question.

<configuration>
    <appender name="file" class="ch.qos.logback.core.FileAppender">
        <file>${log.file}</file>
        <append>false</append>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level
%logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>
    </appender>
   
    <root level="WARN">
        <appender-ref ref="file"/>
    </root>
    <logger name="com.nbnco.csa.analysis.copper.sdc.flink.sink" level="INFO"
additivity="false">
        <appender-ref ref="file"/>
    </logger>
    <logger name="org.apache.flink.streaming.api.functions.source"
level="INFO" additivity="false">
        <appender-ref ref="file"/>
    </logger>
    <logger
name="org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline"
level="ERROR">
        <appender-ref ref="file"/>
    </logger>
</configuration>
 



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-4]

Hi Averell,

I think I have answered your question previously [1]. The bottom line is that
the error is logged on INFO level in the ExecutionGraph [2]. However, your
effective log level (of the root logger) is WARN. The log levels are ordered
as follows [3]:
   
    TRACE < DEBUG < INFO <  WARN < ERROR

It follows that, in your case, log requests to the root logger of level WARN
and ERROR only will appear in the log file – all other levels will be
discarded. It should be enough to set the root log level to INFO, or set the
log level for the org.apache.flink.runtime.executiongraph.ExecutionGraph
logger explicitly to INFO (or lower).

Best,
Gary

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/No-resource-available-error-while-testing-HA-tp25681p25835.html
[2] https://github.com/apache/flink/blob/81acd0a490f3ac40cbb2736189796138ac109dd0/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java#L1363
[3] https://logback.qos.ch/manual/architecture.html

On Fri, Mar 15, 2019 at 4:21 AM Averell <[hidden email]> wrote:

Hi everyone,

I am running Flink in EMR YARN cluster, and when the job failed and
restarted, I could see some logs in the Exception tab of Flink GUI.
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t1586/Screen_Shot_2019-03-15_at_12.png>

I could not find this piece of logs on my cluster's hard-disk - not in TM or
JM logs.

Where can I find this?

Thanks.

Here below is my logback.xml. I'm not sure it has anything to do with my
question.

<configuration>
    <appender name="file" class="ch.qos.logback.core.FileAppender">
        <file>${log.file}</file>
        <append>false</append>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level
%logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="WARN">
        <appender-ref ref="file"/>
    </root>
    <logger name="com.nbnco.csa.analysis.copper.sdc.flink.sink" level="INFO"
additivity="false">
        <appender-ref ref="file"/>
    </logger>
    <logger name="org.apache.flink.streaming.api.functions.source"
level="INFO" additivity="false">
        <appender-ref ref="file"/>
    </logger>
    <logger
name="org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline"
level="ERROR">
        <appender-ref ref="file"/>
    </logger>
</configuration>




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Averell]

Hi Gary,

Thanks a lot for the explanation, and sorry for missing your earlier
message.
I am clear now.

Thanks and regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/