(DEPRECATED) Apache Flink User Mailing List archive.

Help Required for Log4J

Classic

List

Threaded

3 messages Options

Puneet Kinra-2

Help Required for Log4J

I have a use case in which i want to log bad records in the log file. I have configured the log4j

property file is getting generated as well but it also going to flink logs as well i want to detach

it from flink logs want to write to log file.

.Here is configuration

(Note :AMSSource is the custom written adaptor here)

# This affects logging for both user code and Flink

log4j.rootLogger=INFO, file

log4j.logger.amssource=DEBUG, amssourceAppender

# Uncomment this if you want to _only_ change Flink's logging

#log4j.logger.org.apache.flink=INFO

# The following lines keep the log level of common libraries/connectors on

# log level INFO. The root logger does not override this. You have to manually

# change the log levels here.

log4j.logger.akka=INFO

log4j.logger.org.apache.kafka=INFO

log4j.logger.org.apache.hadoop=INFO

log4j.logger.org.apache.zookeeper=INFO

# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender

log4j.appender.file.file=D:\\logs\\flink-log

log4j.appender.file.append=false

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler

log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

#BonusPointAppender

log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender

log4j.appender.bonuspointAppender.MaxFileSize=1024MB

log4j.appender.bonuspointAppender.MaxBackupIndex=10

log4j.appender.bonuspointAppender.Append=true

log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs

#log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd

log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout

log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p (%C %M:%L) %x - %m%n

#AMSSourceAppender

log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender

log4j.appender.amssourceAppender.MaxFileSize=1024MB

log4j.appender.amssourceAppender.MaxBackupIndex=10

log4j.appender.amssourceAppender.Append=true

log4j.appender.amssourceAppender.File=D:\\logs\\flink-bpuser-bonus-amssource.logs

#log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd

log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout

log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p (%C %M:%L) %x - %m%n

Cheers

Puneet Kinra

Mobile:+918800167808 | Skype : [hidden email]

e-mail :[hidden email]

Fabian Hueske-2

Re: Help Required for Log4J

Hi,

TBH, I don't have much experience with logging, but you might want to consider using Side Outputs [1] to route invalid records into a separate stream.

The stream can then separately handled, be written to files or Kafka or wherever.

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/side_output.html

2018-03-20 10:36 GMT+01:00 Puneet Kinra <[hidden email]>:

Hi

I have a use case in which i want to log bad records in the log file. I have configured the log4j
property file is getting generated as well but it also going to flink logs as well i want to detach
it from flink logs want to write to log file.

.Here is configuration
(Note :AMSSource is the custom written adaptor here)

# This affects logging for both user code and Flink
log4j.rootLogger=INFO, file
log4j.logger.amssource=DEBUG, amssourceAppender

# Uncomment this if you want to _only_ change Flink's logging
#log4j.logger.org.apache.flink=INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO

# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=D:\\logs\\flink-log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

#BonusPointAppender
log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender
log4j.appender.bonuspointAppender.MaxFileSize=1024MB
log4j.appender.bonuspointAppender.MaxBackupIndex=10
log4j.appender.bonuspointAppender.Append=true
log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs
#log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p (%C %M:%L) %x - %m%n

#AMSSourceAppender
log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender
log4j.appender.amssourceAppender.MaxFileSize=1024MB
log4j.appender.amssourceAppender.MaxBackupIndex=10
log4j.appender.amssourceAppender.Append=true
log4j.appender.amssourceAppender.File=D:\\logs\\flink-bpuser-bonus-amssource.logs
#log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p (%C %M:%L) %x - %m%n

--
Cheers

Puneet Kinra

Mobile:<a href="tel:+91%2088001%2067808" value="+918800167808" target="_blank">+918800167808 | Skype : [hidden email]
e-mail :[hidden email]