logging question
Locked
logging question
 
|
Re: logging question
|
|
Just a guess, but probably our logging initialisation changes the global
log level (see conf/log4j.properties). DataStream.collect() executes the program along with creating a local Flink "cluster" (if you are testing locally / in an IDE) and initializing logging, among other things. Please comment the first line out and uncomment the following one to read like this: ========== # This affects logging for both user code and Flink #log4j.rootLogger=INFO, file # Uncomment this if you want to _only_ change Flink's logging log4j.logger.org.apache.flink=INFO ========== Nico On 13/01/18 13:52, [hidden email] wrote: > Hello, > I am learning Flink and using the docker image along with the AMIDST > library for this. > Below is a sample task from AMIDST which provides INFO output up until I > reach updateModel(). I pasted the short method as well and wonder what > prevents the Logger from > > //Set-up Flink session > env = ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().disableSysoutLogging(); > Logger LOG = LoggerFactory.getLogger(">>>>> ParallelMLExample"); > > //generate a random dataset > DataFlink<DataInstance> dataFlink = new > DataSetGenerator().generate(env, 1234, 1000, 5, 0); > > //Creates a DAG with the NaiveBayes structure for the random > dataset > DAG dag = > DAGGenerator.getNaiveBayesStructure(dataFlink.getAttributes(), > "DiscreteVar4"); > LOG.info(dag.toString()); > > //Create the Learner object > ParameterLearningAlgorithm learningAlgorithmFlink = new > ParallelMaximumLikelihood(); > > //Learning parameters > learningAlgorithmFlink.setBatchSize(10); > learningAlgorithmFlink.setDAG(dag); > > //Initialize the learning process > learningAlgorithmFlink.initLearning(); > > //Learn from the flink data > LOG.info("BEFORE UPDATEMODEL"); > learningAlgorithmFlink.updateModel(dataFlink); > LOG.info("AFTER UPDATEMODEL"); > > //Print the learnt Bayes Net > BayesianNetwork bn = > learningAlgorithmFlink.getLearntBayesianNetwork(); > LOG.info(bn.toString()); > > > Below is the updateModel method. > > public double updateModel(DataFlink<DataInstance> dataUpdate) { > try { > Configuration config = new Configuration(); > config.setString(BN_NAME, this.dag.getName()); > config.setBytes(EFBN_NAME, > Serialization.serializeObject(efBayesianNetwork)); > > DataSet<DataInstance> dataset = dataUpdate.getDataSet(); > this.sumSS = dataset.map(new SufficientSatisticsMAP()) > .withParameters(config) > .reduce(new SufficientSatisticsReduce()) > .collect().get(0); > > //Add the prior > sumSS.sum(efBayesianNetwork.createInitSufficientStatistics()); > > JobExecutionResult result = > dataset.getExecutionEnvironment().getLastJobExecutionResult(); > > numInstances = > result.getAccumulatorResult(ParallelMaximumLikelihood.COUNTER_NAME+"_"+this.dag.getName()); > > numInstances++;//Initial counts > > }catch(Exception ex){ > throw new UndeclaredThrowableException(ex); > } > > return this.getLogMarginalProbability(); > } > > > Not sure why LOG.info past that method are not output to the console. > TIA > JP |
signature.asc (201 bytes)
Re: logging question
|
|
Hello Nico,
took me a while to respond. Thank you for the comments. I had explored a little more the docker-image and startup scripts. That allowed me to better understand the log4j properties file used but I am still facing this odd behavior. I created a stackoverflow entry for this https://stackoverflow.com/questions/48853497/docker-flink-not-showing-all-log-statements Below, I am just showing the properties file below which I hadn't put on SO. # This affects logging for both user code and Flink log4j.rootLogger=INFO, file, console # Uncomment this if you want to _only_ change Flink's logging log4j.logger.org.apache.flink=OFF # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. log4j.logger.akka=INFO log4j.logger.org.apache.kafka=INFO log4j.logger.org.apache.hadoop=INFO log4j.logger.org.apache.zookeeper=INFO # Log all infos in the given file log4j.appender.file=org.apache.log4j.FileAppender log4j.appender.file.file=${log.file} log4j.appender.file.append=false log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Log all infos to the console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.Target=System.out log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Suppress the irrelevant (wrong) warnings log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF JP On 01/16/2018 10:50 AM, Nico Kruber wrote: > Just a guess, but probably our logging initialisation changes the global > log level (see conf/log4j.properties). DataStream.collect() executes the > program along with creating a local Flink "cluster" (if you are testing > locally / in an IDE) and initializing logging, among other things. > > Please comment the first line out and uncomment the following one to > read like this: > ========== > # This affects logging for both user code and Flink > #log4j.rootLogger=INFO, file > > # Uncomment this if you want to _only_ change Flink's logging > log4j.logger.org.apache.flink=INFO > ========== > > > Nico > > On 13/01/18 13:52, [hidden email] wrote: >> Hello, >> I am learning Flink and using the docker image along with the AMIDST >> library for this. >> Below is a sample task from AMIDST which provides INFO output up until I >> reach updateModel(). I pasted the short method as well and wonder what >> prevents the Logger from >> >> //Set-up Flink session >> env = ExecutionEnvironment.getExecutionEnvironment(); >> env.getConfig().disableSysoutLogging(); >> Logger LOG = LoggerFactory.getLogger(">>>>> ParallelMLExample"); >> >> //generate a random dataset >> DataFlink<DataInstance> dataFlink = new >> DataSetGenerator().generate(env, 1234, 1000, 5, 0); >> >> //Creates a DAG with the NaiveBayes structure for the random >> dataset >> DAG dag = >> DAGGenerator.getNaiveBayesStructure(dataFlink.getAttributes(), >> "DiscreteVar4"); >> LOG.info(dag.toString()); >> >> //Create the Learner object >> ParameterLearningAlgorithm learningAlgorithmFlink = new >> ParallelMaximumLikelihood(); >> >> //Learning parameters >> learningAlgorithmFlink.setBatchSize(10); >> learningAlgorithmFlink.setDAG(dag); >> >> //Initialize the learning process >> learningAlgorithmFlink.initLearning(); >> >> //Learn from the flink data >> LOG.info("BEFORE UPDATEMODEL"); >> learningAlgorithmFlink.updateModel(dataFlink); >> LOG.info("AFTER UPDATEMODEL"); >> >> //Print the learnt Bayes Net >> BayesianNetwork bn = >> learningAlgorithmFlink.getLearntBayesianNetwork(); >> LOG.info(bn.toString()); >> >> >> Below is the updateModel method. >> >> public double updateModel(DataFlink<DataInstance> dataUpdate) { >> try { >> Configuration config = new Configuration(); >> config.setString(BN_NAME, this.dag.getName()); >> config.setBytes(EFBN_NAME, >> Serialization.serializeObject(efBayesianNetwork)); >> >> DataSet<DataInstance> dataset = dataUpdate.getDataSet(); >> this.sumSS = dataset.map(new SufficientSatisticsMAP()) >> .withParameters(config) >> .reduce(new SufficientSatisticsReduce()) >> .collect().get(0); >> >> //Add the prior >> sumSS.sum(efBayesianNetwork.createInitSufficientStatistics()); >> >> JobExecutionResult result = >> dataset.getExecutionEnvironment().getLastJobExecutionResult(); >> >> numInstances = >> result.getAccumulatorResult(ParallelMaximumLikelihood.COUNTER_NAME+"_"+this.dag.getName()); >> >> numInstances++;//Initial counts >> >> }catch(Exception ex){ >> throw new UndeclaredThrowableException(ex); >> } >> >> return this.getLogMarginalProbability(); >> } >> >> >> Not sure why LOG.info past that method are not output to the console. >> TIA >> JP |
Re: logging question
|
|
I'm a bit curious on how you hand your log4j into the docker image for
consumption. On SO you are referring to bin/flink-console.sh but executing Flink in docker is a bit different. Maybe I'm wrong, but looking at the sources of the docker image [1], it will not forward any additional parameters to the docker container via additions to the command starting the docker image. Nico [1] https://github.com/docker-flink/docker-flink/tree/master/1.4/hadoop28-scala_2.11-alpine On 27/02/18 18:25, JP de Vooght wrote: > Hello Nico, > > took me a while to respond. Thank you for the comments. I had explored a > little more the docker-image and startup scripts. That allowed me to > better understand the log4j properties file used but I am still facing > this odd behavior. > > I created a stackoverflow entry for this > > https://stackoverflow.com/questions/48853497/docker-flink-not-showing-all-log-statements > > Below, I am just showing the properties file below which I hadn't put on SO. > > # This affects logging for both user code and Flink > log4j.rootLogger=INFO, file, console > > # Uncomment this if you want to _only_ change Flink's logging > log4j.logger.org.apache.flink=OFF > > # The following lines keep the log level of common libraries/connectors on > # log level INFO. The root logger does not override this. You have to > manually > # change the log levels here. > log4j.logger.akka=INFO > log4j.logger.org.apache.kafka=INFO > log4j.logger.org.apache.hadoop=INFO > log4j.logger.org.apache.zookeeper=INFO > > # Log all infos in the given file > log4j.appender.file=org.apache.log4j.FileAppender > log4j.appender.file.file=${log.file} > log4j.appender.file.append=false > log4j.appender.file.layout=org.apache.log4j.PatternLayout > log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} > %-5p %-60c %x - %m%n > > # Log all infos to the console > log4j.appender.console=org.apache.log4j.ConsoleAppender > log4j.appender.console.Target=System.out > log4j.appender.console.layout=org.apache.log4j.PatternLayout > log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd > HH:mm:ss,SSS} %-5p %-60c %x - %m%n > > # Suppress the irrelevant (wrong) warnings > log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file > log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF > > JP > > > On 01/16/2018 10:50 AM, Nico Kruber wrote: >> Just a guess, but probably our logging initialisation changes the global >> log level (see conf/log4j.properties). DataStream.collect() executes the >> program along with creating a local Flink "cluster" (if you are testing >> locally / in an IDE) and initializing logging, among other things. >> >> Please comment the first line out and uncomment the following one to >> read like this: >> ========== >> # This affects logging for both user code and Flink >> #log4j.rootLogger=INFO, file >> >> # Uncomment this if you want to _only_ change Flink's logging >> log4j.logger.org.apache.flink=INFO >> ========== >> >> >> Nico >> >> On 13/01/18 13:52, [hidden email] wrote: >>> Hello, >>> I am learning Flink and using the docker image along with the AMIDST >>> library for this. >>> Below is a sample task from AMIDST which provides INFO output up until I >>> reach updateModel(). I pasted the short method as well and wonder what >>> prevents the Logger from >>> >>> //Set-up Flink session >>> env = ExecutionEnvironment.getExecutionEnvironment(); >>> env.getConfig().disableSysoutLogging(); >>> Logger LOG = LoggerFactory.getLogger(">>>>> ParallelMLExample"); >>> >>> //generate a random dataset >>> DataFlink<DataInstance> dataFlink = new >>> DataSetGenerator().generate(env, 1234, 1000, 5, 0); >>> >>> //Creates a DAG with the NaiveBayes structure for the random >>> dataset >>> DAG dag = >>> DAGGenerator.getNaiveBayesStructure(dataFlink.getAttributes(), >>> "DiscreteVar4"); >>> LOG.info(dag.toString()); >>> >>> //Create the Learner object >>> ParameterLearningAlgorithm learningAlgorithmFlink = new >>> ParallelMaximumLikelihood(); >>> >>> //Learning parameters >>> learningAlgorithmFlink.setBatchSize(10); >>> learningAlgorithmFlink.setDAG(dag); >>> >>> //Initialize the learning process >>> learningAlgorithmFlink.initLearning(); >>> >>> //Learn from the flink data >>> LOG.info("BEFORE UPDATEMODEL"); >>> learningAlgorithmFlink.updateModel(dataFlink); >>> LOG.info("AFTER UPDATEMODEL"); >>> >>> //Print the learnt Bayes Net >>> BayesianNetwork bn = >>> learningAlgorithmFlink.getLearntBayesianNetwork(); >>> LOG.info(bn.toString()); >>> >>> >>> Below is the updateModel method. >>> >>> public double updateModel(DataFlink<DataInstance> dataUpdate) { >>> try { >>> Configuration config = new Configuration(); >>> config.setString(BN_NAME, this.dag.getName()); >>> config.setBytes(EFBN_NAME, >>> Serialization.serializeObject(efBayesianNetwork)); >>> >>> DataSet<DataInstance> dataset = dataUpdate.getDataSet(); >>> this.sumSS = dataset.map(new SufficientSatisticsMAP()) >>> .withParameters(config) >>> .reduce(new SufficientSatisticsReduce()) >>> .collect().get(0); >>> >>> //Add the prior >>> sumSS.sum(efBayesianNetwork.createInitSufficientStatistics()); >>> >>> JobExecutionResult result = >>> dataset.getExecutionEnvironment().getLastJobExecutionResult(); >>> >>> numInstances = >>> result.getAccumulatorResult(ParallelMaximumLikelihood.COUNTER_NAME+"_"+this.dag.getName()); >>> >>> numInstances++;//Initial counts >>> >>> }catch(Exception ex){ >>> throw new UndeclaredThrowableException(ex); >>> } >>> >>> return this.getLogMarginalProbability(); >>> } >>> >>> >>> Not sure why LOG.info past that method are not output to the console. >>> TIA >>> JP > |
Re: logging question
|
|
Re: logging question
|
|
Re: logging question
|
|
Re: logging question
|
|
| Free forum by Nabble | Edit this page |