logback.xml and logback-yarn.xml rollingpolicy configuration

[Balaji Rajagopalan]

The are two files in the /usr/share/flink/conf directory, and I was trying to do the rolling of application logs which goes to following directory in task nodes. 

/var/log/hadoop-yarn/containers/application_*/container_*/taskmanager.log out err

Changing the logback.xml and  logback-yarn.xml has no effect on the log file size nor any rolling is happening. Any inputs please. 

Here is the configuration I have, anything wrong with the config or the approach. 

<configuration>
   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>${log.file}</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <!-- daily rollover -->

      <fileNamePattern>${log.file}_%d{yyyy-MM-dd}.%i.log</fileNamePattern>

      <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

          <!-- or whenever the file size reaches 50MB -->

        <maxFileSize>50MB</maxFileSize>

      </timeBasedFileNamingAndTriggeringPolicy>

      <!-- keep 30 days' worth of history -->

      <maxHistory>30</maxHistory>

    </rollingPolicy>

       <encoder>
            <charset>UTF-8</charset>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>

    </appender>


    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="ch.qos.logback" level="WARN" />
    <root level="INFO">
        <appender-ref ref="file"/>
        <appender-ref ref="console"/>
    </root>
</configuration>

[Till Rohrmann]

Have you made sure that Flink is using logback [1]?

[1] https://ci.apache.org/projects/flink/flink-docs-master/apis/best_practices.html#using-logback-instead-of-log4j

Cheers,

Till

On Tue, Apr 19, 2016 at 2:01 PM, Balaji Rajagopalan <[hidden email]> wrote:

The are two files in the /usr/share/flink/conf directory, and I was trying to do the rolling of application logs which goes to following directory in task nodes. 

/var/log/hadoop-yarn/containers/application_*/container_*/taskmanager.log out err

Changing the logback.xml and  logback-yarn.xml has no effect on the log file size nor any rolling is happening. Any inputs please. 

Here is the configuration I have, anything wrong with the config or the approach. 

<configuration>
   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>${log.file}</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <!-- daily rollover -->

      <fileNamePattern>${log.file}_%d{yyyy-MM-dd}.%i.log</fileNamePattern>

      <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

          <!-- or whenever the file size reaches 50MB -->

        <maxFileSize>50MB</maxFileSize>

      </timeBasedFileNamingAndTriggeringPolicy>

      <!-- keep 30 days' worth of history -->

      <maxHistory>30</maxHistory>

    </rollingPolicy>

       <encoder>
            <charset>UTF-8</charset>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>

    </appender>


    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="ch.qos.logback" level="WARN" />
    <root level="INFO">
        <appender-ref ref="file"/>
        <appender-ref ref="console"/>
    </root>
</configuration>

[Balaji Rajagopalan]

Thanks Till setting in log4j.properties worked. 

On Tue, Apr 19, 2016 at 8:04 PM, Till Rohrmann <[hidden email]> wrote:

Have you made sure that Flink is using logback [1]?

[1] https://ci.apache.org/projects/flink/flink-docs-master/apis/best_practices.html#using-logback-instead-of-log4j

Cheers,

Till

On Tue, Apr 19, 2016 at 2:01 PM, Balaji Rajagopalan <[hidden email]> wrote:

The are two files in the /usr/share/flink/conf directory, and I was trying to do the rolling of application logs which goes to following directory in task nodes. 

/var/log/hadoop-yarn/containers/application_*/container_*/taskmanager.log out err

Changing the logback.xml and  logback-yarn.xml has no effect on the log file size nor any rolling is happening. Any inputs please. 

Here is the configuration I have, anything wrong with the config or the approach. 

<configuration>
   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>${log.file}</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <!-- daily rollover -->

      <fileNamePattern>${log.file}_%d{yyyy-MM-dd}.%i.log</fileNamePattern>

      <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

          <!-- or whenever the file size reaches 50MB -->

        <maxFileSize>50MB</maxFileSize>

      </timeBasedFileNamingAndTriggeringPolicy>

      <!-- keep 30 days' worth of history -->

      <maxHistory>30</maxHistory>

    </rollingPolicy>

       <encoder>
            <charset>UTF-8</charset>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>

    </appender>


    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="ch.qos.logback" level="WARN" />
    <root level="INFO">
        <appender-ref ref="file"/>
        <appender-ref ref="console"/>
    </root>
</configuration>