Flink log and out files

[Alexander Smirnov]

Hi,

could you please explain the difference between *.log and *.out files in Flink?

What information is supposed to be in each of them?

Is "log" a subset of "out"?

How do I setup rotation with gzipping?

Thank you,

Alex

[vino yang]

Hi Alexander:

.log and .out are different. Usually, the .log file stores the log information output by the log framework. Flink uses slf4j as the log interface and supports log4j and logback configurations. The .out file stores the STDOUT information. This information is usually output by you calling some APIs such as the print sink API.

Thanks, vino.

2018-08-01 23:19 GMT+08:00 Alexander Smirnov <[hidden email]>:

Hi,

could you please explain the difference between *.log and *.out files in Flink?

What information is supposed to be in each of them?

Is "log" a subset of "out"?

How do I setup rotation with gzipping?

Thank you,

Alex

[Andrey Zagrebin]

Hi Alexander,

there is also a doc link where log configuration  is described:

https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/logging.html

You can modify log configuration in conf directory according to logging framework docs.

Cheers,

Andrey

On 1 Aug 2018, at 17:30, vino yang <[hidden email]> wrote:

Hi Alexander:

.log and .out are different. Usually, the .log file stores the log information output by the log framework. Flink uses slf4j as the log interface and supports log4j and logback configurations. The .out file stores the STDOUT information. This information is usually output by you calling some APIs such as the print sink API.

Thanks, vino.

2018-08-01 23:19 GMT+08:00 Alexander Smirnov <[hidden email]>:

Hi,

could you please explain the difference between *.log and *.out files in Flink?

What information is supposed to be in each of them?

Is "log" a subset of "out"?

How do I setup rotation with gzipping?

Thank you,

Alex

[Alexander Smirnov]

thanks guys,

So, is it a correct statement - if my job doesn't write anything to stdout, the "*.out" file should be empty?

for some reason it contains the same info as "log" and much more.

For the "log" files, I can control rotation via log4j configuration, but how do I setup rotation for "out" files? 

Or, how do I disable them at all?

I'm using 1.4.2

Thank you,

Alex

On Wed, Aug 1, 2018 at 7:00 PM Andrey Zagrebin <[hidden email]> wrote:

Hi Alexander,

there is also a doc link where log configuration  is described:

https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/logging.html

You can modify log configuration in conf directory according to logging framework docs.

Cheers,

Andrey

On 1 Aug 2018, at 17:30, vino yang <[hidden email]> wrote:

Hi Alexander:

.log and .out are different. Usually, the .log file stores the log information output by the log framework. Flink uses slf4j as the log interface and supports log4j and logback configurations. The .out file stores the STDOUT information. This information is usually output by you calling some APIs such as the print sink API.

Thanks, vino.

2018-08-01 23:19 GMT+08:00 Alexander Smirnov <[hidden email]>:

Hi,

could you please explain the difference between *.log and *.out files in Flink?

What information is supposed to be in each of them?

Is "log" a subset of "out"?

How do I setup rotation with gzipping?

Thank you,

Alex

[vino yang]

Hi Alexander,

Yes, usually if you configure the log output as a .log file, the .out file will not have content, but sometimes some fatal exceptions, such as JVM exit exceptions, may be printed to the .out file. 

If Flink logs appear in your .out file, and it seems that the format of the log is inconsistent with the format you configured in the configuration file, it is likely that SLF4j detects that there are multiple interface implementation bindings in your jar dependency. 

This will cause the log output to be abnormal, which we have encountered before.

Thanks, vino.

2018-08-02 3:27 GMT+08:00 Alexander Smirnov <[hidden email]>:

thanks guys,

So, is it a correct statement - if my job doesn't write anything to stdout, the "*.out" file should be empty?

for some reason it contains the same info as "log" and much more.

For the "log" files, I can control rotation via log4j configuration, but how do I setup rotation for "out" files? 

Or, how do I disable them at all?

I'm using 1.4.2

Thank you,

Alex

On Wed, Aug 1, 2018 at 7:00 PM Andrey Zagrebin <[hidden email]> wrote:

Hi Alexander,

there is also a doc link where log configuration  is described:

https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/logging.html

You can modify log configuration in conf directory according to logging framework docs.

Cheers,

Andrey

On 1 Aug 2018, at 17:30, vino yang <[hidden email]> wrote:

Hi Alexander:

.log and .out are different. Usually, the .log file stores the log information output by the log framework. Flink uses slf4j as the log interface and supports log4j and logback configurations. The .out file stores the STDOUT information. This information is usually output by you calling some APIs such as the print sink API.

Thanks, vino.

2018-08-01 23:19 GMT+08:00 Alexander Smirnov <[hidden email]>:

Hi,

could you please explain the difference between *.log and *.out files in Flink?

What information is supposed to be in each of them?

Is "log" a subset of "out"?

How do I setup rotation with gzipping?

Thank you,

Alex