Hi to all,
today my code doesn't compile anymore because ExecutionEnvironment doesn't have setConfiguration() anymore..how can I set the following parameters in my unit tests? - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY Best, Flavio |
Hi! Are you on the SNAPSHOT master version? You can pass the configuration to the constructor of the execution environment, or create one via ExecutionEnvironment.createLocalEnvironment(config) or via createRemoteEnvironment(host, port, configuration, jarFiles); The change of the signature was part of an API cleanup for the next release. Sorry for the inconvenience... Stephan On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <[hidden email]> wrote:
|
Yes Stephan!
I usually work with the master version, at least in development ;) Thanks for the quick support! Best, Flavio On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <[hidden email]> wrote:
|
However it could be a good idea to overload also the getExecutionEnvironment() to be able to pass a custom configuration..what do you think?
Otherwise I have to know a priori if I'm working in a local deployment or in a remote one, or check if getExecutionEnvironment() returned an instance of LocalEnvironment/RemoteEnvironment.. On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <[hidden email]> wrote:
|
We can think about that, but I think it may be quite confusing. The configurations actually mean something different for local and remote environments: - For the local environment, the config basically describes the entire Flink cluster setup (for the local execution cluster in the background) - For the remote environment, the config describes the parameters for the client that connects to the cluster (akka paramters, optimizer parameters, ...), but not parameters of the cluster itself (like taskmanager slots and memory). Greetings, Stephan On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <[hidden email]> wrote:
|
That makes sense: what can be configured should be differentiated between local and remote envs (obviously this is a minor issue/improvement)
Thanks again, Flavio On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <[hidden email]> wrote:
|
Hi Fabian and Stephan, back to work :)
I finally managed to find the problem of the parallelism encountered by my colleague! Basically that was introduced by this API change. Before I was using env.setConfiguration() to merge the default params with some custom ones. Now, after the API change I was using the following code: ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); if (env instanceof LocalEnvironment) { Configuration c = new Configuration(); c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR); c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR); c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2); c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4); env = ExecutionEnvironment.createLocalEnvironment(c); } However, the first env and the reassigned one doesn't behave in the same manner. If I don't reassign env I have parallelism=8, otherwise it's 1 :( Am I using the wrong APIs or the execution environment doesn't allow now to configure such parameters anymore? Thanks in advance, Flavio On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <[hidden email]> wrote:
|
Hi Flavio! ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment() by default picks up the number of cores as the parallelism, while the manual environments do not do that. You can still set it manually set the parallelism "env.setParallelism(Runtime.getRuntime().availableProcessors());" I would not configure the slots for the local execution, they should be automatically configured based on the max parallelism. Greetings, Stephan On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <[hidden email]> wrote:
|
of course,I tried to configure the task slot during a debug test and I forgot to remove it.. On 14 Oct 2015 18:35, "Stephan Ewen" <[hidden email]> wrote:
|
I think it's not a nice solution to check for the type of the returned execution environment to determine whether it is a local or a remote execution environment. Wouldn't it be better to add a method isLocal() to ExecutionEnvironment? Cheers, Fabian 2015-10-14 19:14 GMT+02:00 Flavio Pompermaier <[hidden email]>:
|
Free forum by Nabble | Edit this page |