Restoring from an external checkpoint / savepoint with a local stream environment

[Kathleen Sharp]

Hi,

is it possible to restore from an external checkpoint / savepoint
while using a local stream environment?

I ask because I want to play around with some concepts from within my
IDE, without building a jar and deploying my job.

Thanks,
Kat

--
_______________________________________________________

Besuchen Sie uns / Meet us:
OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
OPEX Week World Summit: January 23-27, Orlando, USA
CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
Business Transformation and Operational Excellence World Summit: March
21-24, Orlando, USA

Folgen Sie uns / Follow us:
Twitter | Facebook | LinkedIn | Xing | Youtube
________________________________________________________


HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
Geschäftsführer: Dr. Gero Decker, Guido Sachs

[Ufuk Celebi]

Hey Kathleen,

this is not supported "natively" in the APIs but the following is
possible to work around it:

int jobManagerPort = 6123;
int numTaskSlots = 8;
String savepointPath = "/Users/uce/Desktop/savepoint-xAsd";

Configuration config = new Configuration();
config.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerPort);
config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, numTaskSlots);

LocalFlinkMiniCluster cluster = new LocalFlinkMiniCluster(config, false);
cluster.start(true);

// The "remote environment" submits to the local cluster
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createRemoteEnvironment("localhost",
jobManagerPort);

// Create your program as usual
// ...

// Instead of env.execute do the following:

JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));

boolean printUpdates = true;
cluster.submitJobAndWait(jobGraph, printUpdates);

----

We could think about exposing the SavepointSettings to the
StreamGraph. Then you don't need all the boilerplate for this.


On Fri, Feb 3, 2017 at 12:20 PM, Kathleen Sharp
<[hidden email]> wrote:

> Hi,
>
> is it possible to restore from an external checkpoint / savepoint
> while using a local stream environment?
>
> I ask because I want to play around with some concepts from within my
> IDE, without building a jar and deploying my job.
>
> Thanks,
> Kat
>
> --
> _______________________________________________________
>
> Besuchen Sie uns / Meet us:
> OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
> OPEX Week World Summit: January 23-27, Orlando, USA
> CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
> Business Transformation and Operational Excellence World Summit: March
> 21-24, Orlando, USA
>
> Folgen Sie uns / Follow us:
> Twitter | Facebook | LinkedIn | Xing | Youtube
> ________________________________________________________
>
>
> HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
> Geschäftsführer: Dr. Gero Decker, Guido Sachs

[Kathleen Sharp]

Thanks a lot for the detailed answer. From reading the code i
suspected something would be possible, but this would have taken me a
while to figure out.

On Fri, Feb 3, 2017 at 2:08 PM, Ufuk Celebi <[hidden email]> wrote:

> Hey Kathleen,
>
> this is not supported "natively" in the APIs but the following is
> possible to work around it:
>
> int jobManagerPort = 6123;
> int numTaskSlots = 8;
> String savepointPath = "/Users/uce/Desktop/savepoint-xAsd";
>
> Configuration config = new Configuration();
> config.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerPort);
> config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, numTaskSlots);
>
> LocalFlinkMiniCluster cluster = new LocalFlinkMiniCluster(config, false);
> cluster.start(true);
>
> // The "remote environment" submits to the local cluster
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createRemoteEnvironment("localhost",
> jobManagerPort);
>
> // Create your program as usual
> // ...
>
> // Instead of env.execute do the following:
>
> JobGraph jobGraph = env.getStreamGraph().getJobGraph();
> jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
>
> boolean printUpdates = true;
> cluster.submitJobAndWait(jobGraph, printUpdates);
>
> ----
>
> We could think about exposing the SavepointSettings to the
> StreamGraph. Then you don't need all the boilerplate for this.
>
>
> On Fri, Feb 3, 2017 at 12:20 PM, Kathleen Sharp
> <[hidden email]> wrote:
>> Hi,
>>
>> is it possible to restore from an external checkpoint / savepoint
>> while using a local stream environment?
>>
>> I ask because I want to play around with some concepts from within my
>> IDE, without building a jar and deploying my job.
>>
>> Thanks,
>> Kat
>>
>> --
>> _______________________________________________________
>>
>> Besuchen Sie uns / Meet us:
>> OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
>> OPEX Week World Summit: January 23-27, Orlando, USA
>> CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
>> Business Transformation and Operational Excellence World Summit: March
>> 21-24, Orlando, USA
>>
>> Folgen Sie uns / Follow us:
>> Twitter | Facebook | LinkedIn | Xing | Youtube
>> ________________________________________________________
>>
>>
>> HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
>> Geschäftsführer: Dr. Gero Decker, Guido Sachs

--
_______________________________________________________

Besuchen Sie uns / Meet us:
OPEX-Woche DACH: January 17-19, Wiesbaden, Germany
OPEX Week World Summit: January 23-27, Orlando, USA
CIO EDGE Experience: Febuary 22-23, Melbourne, Australia
Business Transformation and Operational Excellence World Summit: March
21-24, Orlando, USA

Folgen Sie uns / Follow us:
Twitter | Facebook | LinkedIn | Xing | Youtube
________________________________________________________


HRB 121584 B Amtsgericht Charlottenburg, Ust-ID: DE265675123
Geschäftsführer: Dr. Gero Decker, Guido Sachs