Question regarding logging capabilities in flink

[Sharma, Samiksha]

Hi,

I was reading this link regarding logging in flink jobs (https://ci.apache.org/projects/flink/flink-docs-master/internals/logging.html) and with modifications to log4j files I am able to see logs in flink/log directory when I run job Standalone or on Yarn, but I was more interested in seeing the logs I added in my job when I see logs on Yarn UI.

Is there a way in Flink that can enable the user to see the logs added in a job on Yarn rather than on client side.?

Thanks

Samiksha

[Maximilian Michels]

Hi,

Flink prints the Yarn application id during deployment of the cluster.
You can then query the logs from Yarn using the `yarn logs
-applicationId <appId>` command.

Please have a look at
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs

Cheers,
Max

On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha
<[hidden email]> wrote:

> Hi,
>
> I was reading this link regarding logging in flink jobs
> (https://ci.apache.org/projects/flink/flink-docs-master/internals/logging.html)
> and with modifications to log4j files I am able to see logs in flink/log
> directory when I run job Standalone or on Yarn, but I was more interested in
> seeing the logs I added in my job when I see logs on Yarn UI.
> Is there a way in Flink that can enable the user to see the logs added in a
> job on Yarn rather than on client side.?
>
> Thanks
> Samiksha
>
>

[VALLEE Charles]

Hi,

http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

-----Message d'origine-----
De : [hidden email] [mailto:[hidden email]]
Envoyé : vendredi 24 juin 2016 13:53
À : [hidden email]
Objet : Re: Question regarding logging capabilities in flink

Hi,

Flink prints the Yarn application id during deployment of the cluster.
You can then query the logs from Yarn using the `yarn logs -applicationId <appId>` command.

Please have a look at
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs

Cheers,
Max

On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha <[hidden email]> wrote:

> Hi,
>
> I was reading this link regarding logging in flink jobs
> (https://ci.apache.org/projects/flink/flink-docs-master/internals/logg
> ing.html) and with modifications to log4j files I am able to see logs
> in flink/log directory when I run job Standalone or on Yarn, but I was
> more interested in seeing the logs I added in my job when I see logs
> on Yarn UI.
> Is there a way in Flink that can enable the user to see the logs added
> in a job on Yarn rather than on client side.?
>
> Thanks
> Samiksha
>
>

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.

[Sharma, Samiksha]

Hi,

I think what I wanted to know was if I add some logging in my flink job, I
do not see them on Yarn.

For example, In the following snippet in WordCount example when I add
Logging, I do not see this in yarn logs when I run the job. However, I see
them as client log on my side but not on yarn. So, is there a way I can
see these on yarn as well along with jobmanager and taskmanager logs.



public class WordCount {

public static void main(String[] args) throws Exception {

Logger LOGGER = LoggerFactory.getLogger(WordCount.class);
LOGGER.info("This message is testing for Logging in this job");
LOGGER.warn("This is just TESTING WARN");



// Checking input parameters
final ParameterTool params = ParameterTool.fromArgs(args);
System.out.println("Usage: WordCount --input <path> --output <path>");




Please let me know if you have any clarification questions.

Thanks
Samiksha Sharma







On 6/24/16, 4:53 AM, "Maximilian Michels" <[hidden email]> wrote:

>Hi,
>
>Flink prints the Yarn application id during deployment of the cluster.
>You can then query the logs from Yarn using the `yarn logs
>-applicationId <appId>` command.
>
>Please have a look at
>https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCom
>mands.html#logs
>
>Cheers,
>Max
>
>On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha
><[hidden email]> wrote:
>> Hi,
>>
>> I was reading this link regarding logging in flink jobs
>>
>>(https://ci.apache.org/projects/flink/flink-docs-master/internals/logging
>>.html)
>> and with modifications to log4j files I am able to see logs in flink/log
>> directory when I run job Standalone or on Yarn, but I was more
>>interested in
>> seeing the logs I added in my job when I see logs on Yarn UI.
>> Is there a way in Flink that can enable the user to see the logs added
>>in a
>> job on Yarn rather than on client side.?
>>
>> Thanks
>> Samiksha
>>
>>

[Sharma, Samiksha]

Just wanted to followup and know if there are any updates regarding this
query?

Thanks

Samiksha Sharma
Engineer
HERE Predictive Analytics

HERE Seattle
701 Pike Street, #2000, Seattle, WA 98101, USA
47° 36' 41" N. 122° 19' 57" W
HERE Maps






On 6/24/16, 12:14 PM, "Sharma, Samiksha" <[hidden email]> wrote:

>Hi,
>
>I think what I wanted to know was if I add some logging in my flink job, I
>do not see them on Yarn.
>
>For example, In the following snippet in WordCount example when I add
>Logging, I do not see this in yarn logs when I run the job. However, I see
>them as client log on my side but not on yarn. So, is there a way I can
>see these on yarn as well along with jobmanager and taskmanager logs.
>
>
>
>public class WordCount {
>
>public static void main(String[] args) throws Exception {
>
>Logger LOGGER = LoggerFactory.getLogger(WordCount.class);
>LOGGER.info("This message is testing for Logging in this job");
>LOGGER.warn("This is just TESTING WARN");
>
>
>
>// Checking input parameters
>final ParameterTool params = ParameterTool.fromArgs(args);
>System.out.println("Usage: WordCount --input <path> --output <path>");
>
>
>
>
>Please let me know if you have any clarification questions.
>
>Thanks
>Samiksha Sharma
>
>
>
>
>
>
>
>On 6/24/16, 4:53 AM, "Maximilian Michels" <[hidden email]> wrote:
>
>>Hi,
>>
>>Flink prints the Yarn application id during deployment of the cluster.
>>You can then query the logs from Yarn using the `yarn logs
>>-applicationId <appId>` command.
>>
>>Please have a look at
>>https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCo
>>m
>>mands.html#logs
>>
>>Cheers,
>>Max
>>
>>On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha
>><[hidden email]> wrote:
>>> Hi,
>>>
>>> I was reading this link regarding logging in flink jobs
>>>
>>>(https://ci.apache.org/projects/flink/flink-docs-master/internals/loggin
>>>g
>>>.html)
>>> and with modifications to log4j files I am able to see logs in
>>>flink/log
>>> directory when I run job Standalone or on Yarn, but I was more
>>>interested in
>>> seeing the logs I added in my job when I see logs on Yarn UI.
>>> Is there a way in Flink that can enable the user to see the logs added
>>>in a
>>> job on Yarn rather than on client side.?
>>>
>>> Thanks
>>> Samiksha
>>>
>>>
>

[Aljoscha Krettek]

Hi,

I think there is no way to get the output from these log statements into the Yarn logs. The reason is that this code is only executed on the client and not in any Yarn context/container. This code is setting up everything for Yarn and then control is handed over so it is executed before the Job is running on Yarn.

I hope that clarifies things a bit.

Best,

Aljoscha

On Wed, 29 Jun 2016 at 22:27 Sharma, Samiksha <[hidden email]> wrote:

Just wanted to followup and know if there are any updates regarding this
query?

Thanks

Samiksha Sharma
Engineer
HERE Predictive Analytics

HERE Seattle
701 Pike Street, #2000, Seattle, WA 98101, USA
47° 36' 41" N. 122° 19' 57" W
HERE Maps






On 6/24/16, 12:14 PM, "Sharma, Samiksha" <[hidden email]> wrote:

>Hi,
>
>I think what I wanted to know was if I add some logging in my flink job, I
>do not see them on Yarn.
>
>For example, In the following snippet in WordCount example when I add
>Logging, I do not see this in yarn logs when I run the job. However, I see
>them as client log on my side but not on yarn. So, is there a way I can
>see these on yarn as well along with jobmanager and taskmanager logs.
>
>
>
>public class WordCount {
>
>public static void main(String[] args) throws Exception {
>
>Logger LOGGER = LoggerFactory.getLogger(WordCount.class);
>LOGGER.info("This message is testing for Logging in this job");
>LOGGER.warn("This is just TESTING WARN");
>
>
>
>// Checking input parameters
>final ParameterTool params = ParameterTool.fromArgs(args);
>System.out.println("Usage: WordCount --input <path> --output <path>");
>
>
>
>
>Please let me know if you have any clarification questions.
>
>Thanks
>Samiksha Sharma
>
>
>
>
>
>
>
>On 6/24/16, 4:53 AM, "Maximilian Michels" <[hidden email]> wrote:
>
>>Hi,
>>
>>Flink prints the Yarn application id during deployment of the cluster.
>>You can then query the logs from Yarn using the `yarn logs
>>-applicationId <appId>` command.
>>
>>Please have a look at
>>https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCo
>>m
>>mands.html#logs
>>
>>Cheers,
>>Max
>>
>>On Thu, Jun 23, 2016 at 7:44 PM, Sharma, Samiksha
>><[hidden email]> wrote:
>>> Hi,
>>>
>>> I was reading this link regarding logging in flink jobs
>>>
>>>(https://ci.apache.org/projects/flink/flink-docs-master/internals/loggin
>>>g
>>>.html)
>>> and with modifications to log4j files I am able to see logs in
>>>flink/log
>>> directory when I run job Standalone or on Yarn, but I was more
>>>interested in
>>> seeing the logs I added in my job when I see logs on Yarn UI.
>>> Is there a way in Flink that can enable the user to see the logs added
>>>in a
>>> job on Yarn rather than on client side.?
>>>
>>> Thanks
>>> Samiksha
>>>
>>>
>