Re: Question regarding logging capabilities in flink
Posted by
Aljoscha Krettek on
Jun 30, 2016; 7:51am
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Question-regarding-logging-capabilities-in-flink-tp7682p7736.html
Hi,
I think there is no way to get the output from these log statements into the Yarn logs. The reason is that this code is only executed on the client and not in any Yarn context/container. This code is setting up everything for Yarn and then control is handed over so it is executed before the Job is running on Yarn.
I hope that clarifies things a bit.
Best,
Aljoscha
Just wanted to followup and know if there are any updates regarding this
query?
Thanks
Samiksha Sharma
Engineer
HERE Predictive Analytics
HERE Seattle
701 Pike Street, #2000, Seattle, WA 98101, USA
47° 36' 41" N. 122° 19' 57" W
HERE Maps
On 6/24/16, 12:14 PM, "Sharma, Samiksha" <[hidden email]> wrote:
>Hi,
>
>I think what I wanted to know was if I add some logging in my flink job, I
>do not see them on Yarn.
>
>For example, In the following snippet in WordCount example when I add
>Logging, I do not see this in yarn logs when I run the job. However, I see
>them as client log on my side but not on yarn. So, is there a way I can
>see these on yarn as well along with jobmanager and taskmanager logs.
>
>
>
>public class WordCount {
>
>public static void main(String[] args) throws Exception {
>
>Logger LOGGER = LoggerFactory.getLogger(WordCount.class);
>LOGGER.info("This message is testing for Logging in this job");
>LOGGER.warn("This is just TESTING WARN");
>
>
>
>// Checking input parameters
>final ParameterTool params = ParameterTool.fromArgs(args);
>System.out.println("Usage: WordCount --input <path> --output <path>");
>
>
>
>
>Please let me know if you have any clarification questions.
>
>Thanks
>Samiksha Sharma
>
>
>
>
>
>
>
>On 6/24/16, 4:53 AM, "Maximilian Michels" <[hidden email]> wrote:
>
>>Hi,
>>
>>Flink prints the Yarn application id during deployment of the cluster.
>>You can then query the logs from Yarn using the `yarn logs
>>-applicationId <appId>` command.
>>
>>Please have a look at
>>https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCo
>>m
>>mands.html#logs
>>
>>Cheers,
>>Max
>>
>>On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha
>><[hidden email]> wrote:
>>> Hi,
>>>
>>> I was reading this link regarding logging in flink jobs
>>>
>>>(https://ci.apache.org/projects/flink/flink-docs-master/internals/loggin
>>>g
>>>.html)
>>> and with modifications to log4j files I am able to see logs in
>>>flink/log
>>> directory when I run job Standalone or on Yarn, but I was more
>>>interested in
>>> seeing the logs I added in my job when I see logs on Yarn UI.
>>> Is there a way in Flink that can enable the user to see the logs added
>>>in a
>>> job on Yarn rather than on client side.?
>>>
>>> Thanks
>>> Samiksha
>>>
>>>
>