Best way for simple logging in jobs?

[LINZ, Arnaud]

Hi,

 

I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…).  It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err has no effect.

 

For now I’m using a “StringBuffer” accumulator but it does not work in streaming apps before v0.10, and only show results at the end.

 

I’ll probably end up using a HDFS logging system but there is maybe a smarter way ?

 

Greetings,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[rmetzger0]

Hi,

Creating a slf4j logger like this:

private static final Logger LOG = LoggerFactory.getLogger(PimpedKafkaSink.class);
Works for me. The messages also end up in the regular YARN logs.
Also system out should end up in YARN actually (when retrieving the logs from the log aggregation).

Regards,

Robert

On Fri, Aug 28, 2015 at 3:55 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,

 

I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…).  It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err has no effect.

 

For now I’m using a “StringBuffer” accumulator but it does not work in streaming apps before v0.10, and only show results at the end.

 

I’ll probably end up using a HDFS logging system but there is maybe a smarter way ?

 

Greetings,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[Stephan Ewen]

@Arnaud

Are you looking for a separate user log file next to the system log file, or would Robert's suggestion work?

On Fri, Aug 28, 2015 at 4:20 PM, Robert Metzger <[hidden email]> wrote:

Hi,

Creating a slf4j logger like this:

private static final Logger LOG = LoggerFactory.getLogger(PimpedKafkaSink.class);
Works for me. The messages also end up in the regular YARN logs.
Also system out should end up in YARN actually (when retrieving the logs from the log aggregation).

Regards,

Robert

On Fri, Aug 28, 2015 at 3:55 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,

 

I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…).  It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err has no effect.

 

For now I’m using a “StringBuffer” accumulator but it does not work in streaming apps before v0.10, and only show results at the end.

 

I’ll probably end up using a HDFS logging system but there is maybe a smarter way ?

 

Greetings,

Arnaud

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[LINZ, Arnaud]

Hi,

For unknown reasons, the stdout/stderr output of my jobs wasn’t retrieved by Yarn. Same thing for slf4j logger : outside local cluster mode, I could not see any trace from the nodes.  I’ve spent a few hours trying to find why, but I gave up.

Since I need “real time” logging & monitoring in the main driver application, I’ve ended up implementing a simple push/pull socket message queue using JeroMQ (pure-Java 0MQ). That allows me to easily publish log & metrology information from the worker nodes to the driver program.

Arnaud

 

 

De : [hidden email] [mailto:[hidden email]] De la part de Stephan Ewen
Envoyé : lundi 31 août 2015 11:05
À : [hidden email]
Objet : Re: Best way for simple logging in jobs?

 

@Arnaud

 

Are you looking for a separate user log file next to the system log file, or would Robert's suggestion work?

 

On Fri, Aug 28, 2015 at 4:20 PM, Robert Metzger <[hidden email]> wrote:

Hi,

 

Creating a slf4j logger like this:

private static final Logger LOG = LoggerFactory.getLogger(PimpedKafkaSink.class);

Works for me. The messages also end up in the regular YARN logs.

Also system out should end up in YARN actually (when retrieving the logs from the log aggregation).

Regards,

Robert

 

On Fri, Aug 28, 2015 at 3:55 PM, LINZ, Arnaud <[hidden email]> wrote:

Hi,

 

I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers, sinks…).  It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err has no effect.

 

For now I’m using a “StringBuffer” accumulator but it does not work in streaming apps before v0.10, and only show results at the end.

 

I’ll probably end up using a HDFS logging system but there is maybe a smarter way ?

 

Greetings,

Arnaud

 

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.