Converting non-mini-batch to mini-batch from checkpoint or savepoint

[Rex Fenley]

Hello,

Is it safe to convert a non-mini-batch job to a mini-batch job when restoring from a checkpoint or a savepoint?

Thanks

--

[Rex Fenley]

Just tested this and I couldn't restore from a savepoint. If I do a new job from scratch, can I tune the minibatch parameters and restore from a savepoint without having to make yet another brand new job?

Thanks

On Wed, Jan 20, 2021 at 12:43 PM Rex Fenley <[hidden email]> wrote:

Hello,

Is it safe to convert a non-mini-batch job to a mini-batch job when restoring from a checkpoint or a savepoint?

Thanks

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Dawid Wysakowicz-2]

I am pulling in Jark and Godfrey who are more familiar with the internals of the planner.

On 21/01/2021 01:43, Rex Fenley wrote:

Just tested this and I couldn't restore from a savepoint. If I do a new job from scratch, can I tune the minibatch parameters and restore from a savepoint without having to make yet another brand new job?

Thanks

On Wed, Jan 20, 2021 at 12:43 PM Rex Fenley <[hidden email]> wrote:

Hello,

Is it safe to convert a non-mini-batch job to a mini-batch job when restoring from a checkpoint or a savepoint?

Thanks

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Jark Wu-3]

Hi Rex,

Currently, it is not state compatible, because we will add a new node called MiniBatchAssigner after the source which changes the JobGraph , thus uid is different.

Best,

Jark

On Tue, 26 Jan 2021 at 18:33, Dawid Wysakowicz <[hidden email]> wrote:

I am pulling in Jark and Godfrey who are more familiar with the internals of the planner.

On 21/01/2021 01:43, Rex Fenley wrote:

Just tested this and I couldn't restore from a savepoint. If I do a new job from scratch, can I tune the minibatch parameters and restore from a savepoint without having to make yet another brand new job?

Thanks

On Wed, Jan 20, 2021 at 12:43 PM Rex Fenley <[hidden email]> wrote:

Hello,

Is it safe to convert a non-mini-batch job to a mini-batch job when restoring from a checkpoint or a savepoint?

Thanks

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[Rex Fenley]

Thanks for the clarification.

On Wed, Jan 27, 2021 at 7:24 PM Jark Wu <[hidden email]> wrote:

Hi Rex,

Currently, it is not state compatible, because we will add a new node called MiniBatchAssigner after the source which changes the JobGraph , thus uid is different.

Best,

Jark

On Tue, 26 Jan 2021 at 18:33, Dawid Wysakowicz <[hidden email]> wrote:

I am pulling in Jark and Godfrey who are more familiar with the internals of the planner.

On 21/01/2021 01:43, Rex Fenley wrote:

Just tested this and I couldn't restore from a savepoint. If I do a new job from scratch, can I tune the minibatch parameters and restore from a savepoint without having to make yet another brand new job?

Thanks

On Wed, Jan 20, 2021 at 12:43 PM Rex Fenley <[hidden email]> wrote:

Hello,

Is it safe to convert a non-mini-batch job to a mini-batch job when restoring from a checkpoint or a savepoint?

Thanks

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--