Re: RocksDB default logging configuration

Posted by Bajaj, Abhinav on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/RocksDB-default-logging-configuration-tp34502p34598.html

Bumping this one again to catch some attention.

 

From: "Bajaj, Abhinav" <[hidden email]>
Date: Monday, April 20, 2020 at 3:23 PM
To: "[hidden email]" <[hidden email]>
Subject: RocksDB default logging configuration

 

Hi,

 

Some of our teams ran into the disk space issues because of RocksDB default logging configuration - FLINK-15068.

It seems the workaround suggested uses the OptionsFactory to set some of the parameters from inside the job.

 

Since we provision the Flink cluster(version 1.7.1) for the teams, we control the RocksDB statebackend configuration from flink-conf.yaml.

And it seems there isn’t any related RocksDB configuration to set in flink-conf.yaml.

 

Is there a way for the job developer to retrieve the default statebackend information from the cluster in the job and set the DBOptions on top of it?

 

Appreciate the help!

 

~ Abhinav Bajaj

 

PS:  Sharing below snippet as desired option if possible -

 

StreamExecutionEnvironment streamExecEnv = StreamExecutionEnvironment.getExecutionEnvironment();

StateBackend stateBackend = streamExecEnv.getDefaultStateBackend();

stateBackend.setOptions(new OptionsFactory() {

@Override
public DBOptions createDBOptions(DBOptions dbOptions) {
  dbOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
  dbOptions.setMaxLogFileSize(1024 * 1024)
  return dbOptions;
}

@Override
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions columnFamilyOptions) {
  return columnFamilyOptions;
}

});