http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-logging-issue-with-logback-tp31928p32017.html
Interesting that we came across this problem at the same time. We have
job. Of course, this is just meant to be a workaround.
I think an acceptable solution is to always print upon execution. For
> A quick update. The suppression of stdout/stderr actually might soon be
> dropped, see:
https://issues.apache.org/jira/browse/FLINK-15504>
> Best,
>
> Dawid
>
> On 07/01/2020 07:17, Yang Wang wrote:
>> Hi Bajaj,
>>
>> I have tested just as you say, and find that the logs in the user
>> class could not show up when
>> using ConsoleAppender. If using FileAppender instead, everything goes
>> well.
>>
>> It is so weird and i have no idea how to debug it.
>> Best,
>> Yang
>>
>> Bajaj, Abhinav <
[hidden email]
>> <mailto:
[hidden email]>> 于2020年1月7日周二 上午4:28写道：
>>
>> Hi,
>>
>> Thanks much for the responses.
>>
>> Let me add some more details and clarify my question.
>>
>> _Setup_
>>
>> * I used the WikipediaAnalysis example and added a log in main
>> method.
>>
>> ……
>>
>> public static void main(String[] args) throws Exception {
>> StreamExecutionEnvironment see =
>> StreamExecutionEnvironment./getExecutionEnvironment/();
>> /LOG/.info("Info log for test");
>>
>> DataStream<WikipediaEditEvent> edits = see.addSource(new
>> WikipediaEditsSource());
>>
>> ……
>>
>> * I am using the Flink 1.7.1 distribution and starting
>> jobmanager and taskmanager locally using the below commands –
>> o ./bin/jobmanager.sh start-foreground
>> o ./bin/taskmanager.sh start-foreground
>> o Both jobmanager and taskmanager log in the console now
>> o JVM options are correctly set and verified from jobmanager
>> & taskmanager logs
>>
>> * I submit the WikipediaAnalysis job from Flink dashboard and
>> checked the jobmanager logs
>>
>> _Run 1_: Flink is using the default log4j logging
>>
>> * Jobmanager logs the added info log from the job
>> o 2020-01-06 11:55:37,422 INFO wikiedits.WikipediaAnalysis -
>> Info log for test
>>
>> _Run 2_: Flink is setup to use logback as suggested in Flink
>> documentation here
>> <
https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster>
>>
>> * Jobmanger does not log the added info log from the job
>>
>> So, it seems there is a logging behavior difference between using
>> log4j & logback in Flink.
>>
>> Is this expected or a known difference?
>>
>> Thanks again,
>>
>> Abhinav Bajaj
>>
>> _PS_: Ahh. I see how my email was confusing the first time.
>> Hopefully this one is better :P
>>
>> *From: *Dawid Wysakowicz <
[hidden email]
>> <mailto:
[hidden email]>>
>> *Date: *Monday, January 6, 2020 at 5:13 AM
>> *Cc: *"Bajaj, Abhinav" <
[hidden email]
>> <mailto:
[hidden email]>>, "
[hidden email]
>> <mailto:
[hidden email]>" <
[hidden email]
>> <mailto:
[hidden email]>>
>> *Subject: *Re: Flink logging issue with logback
>>
>> Hi Bajaj,
>>
>> I am not entirely sure what is the actual issue you are seeking
>> help, but let me comment on your observations.
>>
>> Ad. 1
>>
>> If you log to the console from the main method this is an expected
>> behavior in both cases (log4j, logback). The std out is being
>> overwritten for the execution of the main method
>>
>> If you log to a file logs should work in both cases. I checked
>> that myself and actually the logs appeared in the jobmanager logs
>> as long as they are executed before the env.execute(). I observed
>> though weird behavior of the Web UI, as it does not always update
>> the logs that are displayed. How did you check the logs? If you
>> checked through the Web UI could you try to check the file directly?
>>
>> Ad. 2 Yes this is expected. Operators are executed on taskmanager
>> and that's why they log there.
>>
>> Ad. 3 See Ad. 1
>>
>> Best,
>>
>> Dawid
>>
>> On 06/01/2020 07:07, vino yang wrote:
>>
>> Hi Bajaj,
>>
>> >> Logs from main method(outside of job graph) do not show up
>> in jobmanager logs.
>>
>> IMO, it's normal phenomena.
>>
>> Other ideas, please check the JVM options mentioned by Yang.
>>
>> Best,
>>
>> Vino
>>
>> Yang Wang <
[hidden email]
>> <mailto:
[hidden email]>> 于2020年1月6日周一 上午11:18写道：
>>
>> Hi Bajaj, Abhinav,
>>
>> Could you share the start-command of jobmanager and
>> taskmanager. If it is started correctly, we
>>
>> will have a the following jvm options.
>>
>> -Dlog.file=/path/of/taskmanager.log
>> -Dlogback.configurationFile=file:///path/of/logback.xml
>>
>> Best,
>>
>> Yang
>>
>> Bajaj, Abhinav <
[hidden email]
>> <mailto:
[hidden email]>> 于2020年1月4日周六 上午
>> 7:23写道：
>>
>> Hi,
>>
>> I am investigating a logging issue with Flink.
>>
>> _Setup_
>>
>> * Using Flink-1.7.1 using logback as suggested in
>> Flink documentation here
>> <
https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster>.
>> * Submitting the Flink job from the Flink dashboard.
>>
>> _Observations_
>>
>> * Logs from main method(outside of job graph) do not
>> show up in jobmanager logs.
>> * Logs from the operators like map or custom
>> operators do show up in the taskmanager logs.
>> * Logs from main method do show up in jobmanager
>> logs when using log4j in place of logback.
>>
>> Has anyone else noticed similar behavior or is this a
>> known issue with logback integration in Flink?
>>
>> Any suggestions on potential workaround or fix?
>>
>> Appreciate your time and help.
>>
>> ~ Abhinav Bajaj
>>