Logging when building and testing Flink

[Juha Mynttinen-2]

Hey there,

I noticed that when building and testing Flink itself, logging seems to be non-existing or very quiet. 

I had a look at the logging conf files (such as flink-tests/src/test/resources/log4j2-test.properties) and the pattern seems to be that the logging is turned off in tests. At least it looks like that to me in the conf files and I couldn't find any logs.

I understand that excessive logging can cause flooding of build logs, when logging to stderr and this can be a problem in e.g. the CI environment. Is there some other rationale why logging is disabled?

It'd be super nice if one could somehow globally enable test logging and log to e.g. rolling log files. Is there such a mechanism?

Missing logs can be a problem with testing locally. If you hit such an issue when running the tests locally that the logs would help to fix, there are no logs and you need to enable logging and re-run the tests. This is not great. And if the bug is a "Heisenbug" you might not find the bug on the re-run.

Regards,

Juha

[Dawid Wysakowicz-2]

You should be able to globally override the configuration file used by
surefire plugin which executes tests like this:

mvn '-Dlog4j.configuration=[path]/log4j2-on.properties' clean install

Bear in mind there is a minor bug in our surefire configuration now:
https://issues.apache.org/jira/browse/FLINK-19900

Best,

Dawid

On 23/10/2020 09:48, Juha Mynttinen wrote:

> Hey there,
>
> I noticed that when building and testing Flink itself, logging seems
> to be non-existing or very quiet. 
>
> I had a look at the logging conf files (such
> as flink-tests/src/test/resources/log4j2-test.properties) and the
> pattern seems to be that the logging is turned off in tests. At least
> it looks like that to me in the conf files and I couldn't find any logs.
>
> I understand that excessive logging can cause flooding of build logs,
> when logging to stderr and this can be a problem in e.g. the CI
> environment. Is there some other rationale why logging is disabled?
>
> It'd be super nice if one could somehow globally enable test logging
> and log to e.g. rolling log files. Is there such a mechanism?
>
> Missing logs can be a problem with testing locally. If you hit such an
> issue when running the tests locally that the logs would help to fix,
> there are no logs and you need to enable logging and re-run the tests.
> This is not great. And if the bug is a "Heisenbug" you might not find
> the bug on the re-run.
>
> Regards,
> Juha

[Dawid Wysakowicz-2]

Small correction to my previous email. The previously mentioned problem
is actually not a problem. You can just pass the log4j.configurationFile
explicitly:

mvn '-Dlog4j.configurationFile=[path]/log4j2-on.properties' clean install

Best,

Dawid

On 23/10/2020 09:48, Juha Mynttinen wrote:

> Hey there,
>
> I noticed that when building and testing Flink itself, logging seems
> to be non-existing or very quiet. 
>
> I had a look at the logging conf files (such
> as flink-tests/src/test/resources/log4j2-test.properties) and the
> pattern seems to be that the logging is turned off in tests. At least
> it looks like that to me in the conf files and I couldn't find any logs.
>
> I understand that excessive logging can cause flooding of build logs,
> when logging to stderr and this can be a problem in e.g. the CI
> environment. Is there some other rationale why logging is disabled?
>
> It'd be super nice if one could somehow globally enable test logging
> and log to e.g. rolling log files. Is there such a mechanism?
>
> Missing logs can be a problem with testing locally. If you hit such an
> issue when running the tests locally that the logs would help to fix,
> there are no logs and you need to enable logging and re-run the tests.
> This is not great. And if the bug is a "Heisenbug" you might not find
> the bug on the re-run.
>
> Regards,
> Juha

[Juha Mynttinen-2]

Very nice Dawid, thanks! I'll give this a try next time I run the tests.

Regards,

Juha

El vie., 30 oct. 2020 a las 13:13, Dawid Wysakowicz (<[hidden email]>) escribió:

Small correction to my previous email. The previously mentioned problem
is actually not a problem. You can just pass the log4j.configurationFile
explicitly:

mvn '-Dlog4j.configurationFile=[path]/log4j2-on.properties' clean install

Best,

Dawid

On 23/10/2020 09:48, Juha Mynttinen wrote:
> Hey there,
>
> I noticed that when building and testing Flink itself, logging seems
> to be non-existing or very quiet. 
>
> I had a look at the logging conf files (such
> as flink-tests/src/test/resources/log4j2-test.properties) and the
> pattern seems to be that the logging is turned off in tests. At least
> it looks like that to me in the conf files and I couldn't find any logs.
>
> I understand that excessive logging can cause flooding of build logs,
> when logging to stderr and this can be a problem in e.g. the CI
> environment. Is there some other rationale why logging is disabled?
>
> It'd be super nice if one could somehow globally enable test logging
> and log to e.g. rolling log files. Is there such a mechanism?
>
> Missing logs can be a problem with testing locally. If you hit such an
> issue when running the tests locally that the logs would help to fix,
> there are no logs and you need to enable logging and re-run the tests.
> This is not great. And if the bug is a "Heisenbug" you might not find
> the bug on the re-run.
>
> Regards,
> Juha