state.checkpoints.dir not configured

[Plamen Paskov]

Hi,
I'm trying to enable externalized checkpoints like this:

env.enableCheckpointing(1000);
CheckpointConfig checkpointConfig = env.getCheckpointConfig();
checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.setStateBackend(new FsStateBackend("file:///tmp/flink-checkpoints-data/", true));

in flink-conf.yaml i set:
state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/

but when i run the application i get this error: java.lang.IllegalStateException: CheckpointConfig says to persist periodic checkpoints, but no checkpoint directory has been configured. You can configure configure one via key 'state.checkpoints.dir'.

Any suggestions?

Thanks

[Ufuk Celebi]

When the JobManager/TaskManager are starting up they log what config
they are loading. Look for lines like

"Loading configuration property: {}, {}"

Do you find the required configuration as part of these messages?

– Ufuk


On Tue, Dec 19, 2017 at 3:45 PM, Plamen Paskov
<[hidden email]> wrote:

> Hi,
> I'm trying to enable externalized checkpoints like this:
>
> env.enableCheckpointing(1000);
> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
> checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> env.setStateBackend(new
> FsStateBackend("file:///tmp/flink-checkpoints-data/", true));
>
> in flink-conf.yaml i set:
> state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/
>
> but when i run the application i get this error:
> java.lang.IllegalStateException: CheckpointConfig says to persist periodic
> checkpoints, but no checkpoint directory has been configured. You can
> configure configure one via key 'state.checkpoints.dir'.
>
> Any suggestions?
>
> Thanks

[Plamen Paskov]

I inspected the log as you suggest and found that 6123 port was used by
another process. I free the port and restarted the job manager. Now
everything looks fine. The error message is little misleading as the
real cause is that 6123 is already bind but it says that
state.checkpoints.dir is not set.

Thanks


On 19.12.2017 17:55, Ufuk Celebi wrote:

> When the JobManager/TaskManager are starting up they log what config
> they are loading. Look for lines like
>
> "Loading configuration property: {}, {}"
>
> Do you find the required configuration as part of these messages?
>
> – Ufuk
>
>
> On Tue, Dec 19, 2017 at 3:45 PM, Plamen Paskov
> <[hidden email]> wrote:
>> Hi,
>> I'm trying to enable externalized checkpoints like this:
>>
>> env.enableCheckpointing(1000);
>> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
>> checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>> env.setStateBackend(new
>> FsStateBackend("file:///tmp/flink-checkpoints-data/", true));
>>
>> in flink-conf.yaml i set:
>> state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/
>>
>> but when i run the application i get this error:
>> java.lang.IllegalStateException: CheckpointConfig says to persist periodic
>> checkpoints, but no checkpoint directory has been configured. You can
>> configure configure one via key 'state.checkpoints.dir'.
>>
>> Any suggestions?
>>
>> Thanks

[Ufuk Celebi]

Could you please share the complete logs of the initial failure? What
you describe in your second email should not happen ;-) If the
JobManager cannot bind to the port it should simply die and not
complain about checkpoint configuration.

– Ufuk

On Thu, Dec 21, 2017 at 1:21 PM, Plamen Paskov
<[hidden email]> wrote:

> I inspected the log as you suggest and found that 6123 port was used by
> another process. I free the port and restarted the job manager. Now
> everything looks fine. The error message is little misleading as the real
> cause is that 6123 is already bind but it says that state.checkpoints.dir is
> not set.
>
> Thanks
>
>
>
> On 19.12.2017 17:55, Ufuk Celebi wrote:
>>
>> When the JobManager/TaskManager are starting up they log what config
>> they are loading. Look for lines like
>>
>> "Loading configuration property: {}, {}"
>>
>> Do you find the required configuration as part of these messages?
>>
>> – Ufuk
>>
>>
>> On Tue, Dec 19, 2017 at 3:45 PM, Plamen Paskov
>> <[hidden email]> wrote:
>>>
>>> Hi,
>>> I'm trying to enable externalized checkpoints like this:
>>>
>>> env.enableCheckpointing(1000);
>>> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
>>>
>>> checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>>> env.setStateBackend(new
>>> FsStateBackend("file:///tmp/flink-checkpoints-data/", true));
>>>
>>> in flink-conf.yaml i set:
>>> state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/
>>>
>>> but when i run the application i get this error:
>>> java.lang.IllegalStateException: CheckpointConfig says to persist
>>> periodic
>>> checkpoints, but no checkpoint directory has been configured. You can
>>> configure configure one via key 'state.checkpoints.dir'.
>>>
>>> Any suggestions?
>>>
>>> Thanks
>
>

[Plamen Paskov]

I'm sorry but i already cleaned up the logs. If i encounter the same
error again i will let you know


On 21.12.2017 17:12, Ufuk Celebi wrote:

> Could you please share the complete logs of the initial failure? What
> you describe in your second email should not happen ;-) If the
> JobManager cannot bind to the port it should simply die and not
> complain about checkpoint configuration.
>
> – Ufuk
>
> On Thu, Dec 21, 2017 at 1:21 PM, Plamen Paskov
> <[hidden email]> wrote:
>> I inspected the log as you suggest and found that 6123 port was used by
>> another process. I free the port and restarted the job manager. Now
>> everything looks fine. The error message is little misleading as the real
>> cause is that 6123 is already bind but it says that state.checkpoints.dir is
>> not set.
>>
>> Thanks
>>
>>
>>
>> On 19.12.2017 17:55, Ufuk Celebi wrote:
>>> When the JobManager/TaskManager are starting up they log what config
>>> they are loading. Look for lines like
>>>
>>> "Loading configuration property: {}, {}"
>>>
>>> Do you find the required configuration as part of these messages?
>>>
>>> – Ufuk
>>>
>>>
>>> On Tue, Dec 19, 2017 at 3:45 PM, Plamen Paskov
>>> <[hidden email]> wrote:
>>>> Hi,
>>>> I'm trying to enable externalized checkpoints like this:
>>>>
>>>> env.enableCheckpointing(1000);
>>>> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
>>>>
>>>> checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>>>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>>>> env.setStateBackend(new
>>>> FsStateBackend("file:///tmp/flink-checkpoints-data/", true));
>>>>
>>>> in flink-conf.yaml i set:
>>>> state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/
>>>>
>>>> but when i run the application i get this error:
>>>> java.lang.IllegalStateException: CheckpointConfig says to persist
>>>> periodic
>>>> checkpoints, but no checkpoint directory has been configured. You can
>>>> configure configure one via key 'state.checkpoints.dir'.
>>>>
>>>> Any suggestions?
>>>>
>>>> Thanks
>>

[Aljoscha Krettek]

Hi,

I think what might have happened is that you had an earlier JobManager still running that had the old configuration loaded. Then you tried starting a new JobManager. Could that be the case?

Best,
Aljoscha

> On 21. Dec 2017, at 16:34, Plamen Paskov <[hidden email]> wrote:
>
> I'm sorry but i already cleaned up the logs. If i encounter the same error again i will let you know
>
>
> On 21.12.2017 17:12, Ufuk Celebi wrote:
>> Could you please share the complete logs of the initial failure? What
>> you describe in your second email should not happen ;-) If the
>> JobManager cannot bind to the port it should simply die and not
>> complain about checkpoint configuration.
>>
>> – Ufuk
>>
>> On Thu, Dec 21, 2017 at 1:21 PM, Plamen Paskov
>> <[hidden email]> wrote:
>>> I inspected the log as you suggest and found that 6123 port was used by
>>> another process. I free the port and restarted the job manager. Now
>>> everything looks fine. The error message is little misleading as the real
>>> cause is that 6123 is already bind but it says that state.checkpoints.dir is
>>> not set.
>>>
>>> Thanks
>>>
>>>
>>>
>>> On 19.12.2017 17:55, Ufuk Celebi wrote:
>>>> When the JobManager/TaskManager are starting up they log what config
>>>> they are loading. Look for lines like
>>>>
>>>> "Loading configuration property: {}, {}"
>>>>
>>>> Do you find the required configuration as part of these messages?
>>>>
>>>> – Ufuk
>>>>
>>>>
>>>> On Tue, Dec 19, 2017 at 3:45 PM, Plamen Paskov
>>>> <[hidden email]> wrote:
>>>>> Hi,
>>>>> I'm trying to enable externalized checkpoints like this:
>>>>>
>>>>> env.enableCheckpointing(1000);
>>>>> CheckpointConfig checkpointConfig = env.getCheckpointConfig();
>>>>>
>>>>> checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>>>>> checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>>>>> env.setStateBackend(new
>>>>> FsStateBackend("file:///tmp/flink-checkpoints-data/", true));
>>>>>
>>>>> in flink-conf.yaml i set:
>>>>> state.checkpoints.dir: file:///tmp/flink-checkpoints-meta/
>>>>>
>>>>> but when i run the application i get this error:
>>>>> java.lang.IllegalStateException: CheckpointConfig says to persist
>>>>> periodic
>>>>> checkpoints, but no checkpoint directory has been configured. You can
>>>>> configure configure one via key 'state.checkpoints.dir'.
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks
>>>
>