(DEPRECATED) Apache Flink User Mailing List archive.

Debug job execution from savepoint

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
Haddadi Manuel
Reply | Threaded
Open this post in threaded view
♦
♦
|

Debug job execution from savepoint

Haddadi Manuel

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Debug job execution from savepoint

Fabian Hueske-2
Hi Manuel,

I had a look and couldn't find a way to do it.
However, this sounds like a very useful feature to me.

Would you mind creating a Jira issue [1] for that?

Thanks, Fabian


2018-06-18 16:23 GMT+02:00 Haddadi Manuel <[hidden email]>:

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Haddadi Manuel
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Debug job execution from savepoint

Haddadi Manuel

Hi Fabian,


thanks for your response. Here is the Jira issue : https://issues.apache.org/jira/browse/FLINK-9644


Would you have any devlopper tips on how testing my intializeState methods to validate that my savepoints are corrects ?


Thank you again. Regards,


Manuel

De : Fabian Hueske <[hidden email]>
Envoyé : jeudi 21 juin 2018 10:33:10
À : Haddadi Manuel
Cc : [hidden email]
Objet : Re: Debug job execution from savepoint
 
Hi Manuel,

I had a look and couldn't find a way to do it.
However, this sounds like a very useful feature to me.

Would you mind creating a Jira issue [1] for that?

Thanks, Fabian


2018-06-18 16:23 GMT+02:00 Haddadi Manuel <[hidden email]>:

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Debug job execution from savepoint

Tzu-Li (Gordon) Tai
Hi,

The tests in Flink uses a `AbstractStreamOperatorTestHarness` that allows wrapping an operator, input elements into the operator, getting the outputs, and also snapshotting / restoring operator state.
I’m not sure of your specific case, but in general that test harness utility can be used to test the operator’s output and state update logic for a given set of input records.

Cheers,
Gordon

On 22 June 2018 at 11:23:28 AM, Haddadi Manuel ([hidden email]) wrote:

Hi Fabian,


thanks for your response. Here is the Jira issue : https://issues.apache.org/jira/browse/FLINK-9644


Would you have any devlopper tips on how testing my intializeState methods to validate that my savepoints are corrects ?


Thank you again. Regards,


Manuel

De : Fabian Hueske <[hidden email]>
Envoyé : jeudi 21 juin 2018 10:33:10
À : Haddadi Manuel
Cc : [hidden email]
Objet : Re: Debug job execution from savepoint
 
Hi Manuel,

I had a look and couldn't find a way to do it.
However, this sounds like a very useful feature to me.

Would you mind creating a Jira issue [1] for that?

Thanks, Fabian


2018-06-18 16:23 GMT+02:00 Haddadi Manuel <[hidden email]>:

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Debug job execution from savepoint

Chesnay Schepler
Similarly, the AbstractOperatorRestoreTestBase shows how to start en embedded cluster and submit a job from a savepoint.
see
* #restoreJob
* #createJobGraph
* MiniClusterResource

On 22.06.2018 13:53, Tzu-Li (Gordon) Tai wrote:
Hi,

The tests in Flink uses a `AbstractStreamOperatorTestHarness` that allows wrapping an operator, input elements into the operator, getting the outputs, and also snapshotting / restoring operator state.
I’m not sure of your specific case, but in general that test harness utility can be used to test the operator’s output and state update logic for a given set of input records.

Cheers,
Gordon

On 22 June 2018 at 11:23:28 AM, Haddadi Manuel ([hidden email]) wrote:

Hi Fabian,


thanks for your response. Here is the Jira issue : https://issues.apache.org/jira/browse/FLINK-9644


Would you have any devlopper tips on how testing my intializeState methods to validate that my savepoints are corrects ?


Thank you again. Regards,


Manuel

De : Fabian Hueske [hidden email]
Envoyé : jeudi 21 juin 2018 10:33:10
À : Haddadi Manuel
Cc : [hidden email]
Objet : Re: Debug job execution from savepoint
 
Hi Manuel,

I had a look and couldn't find a way to do it.
However, this sounds like a very useful feature to me.

Would you mind creating a Jira issue [1] for that?

Thanks, Fabian


2018-06-18 16:23 GMT+02:00 Haddadi Manuel <[hidden email]>:

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Haddadi Manuel
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Debug job execution from savepoint

Haddadi Manuel

Great !

Thanks for your help. I think I will have all the functionalities I need for my use case with these test classes.

Regards,

Manuel

De : Chesnay Schepler <[hidden email]>
Envoyé : vendredi 22 juin 2018 15:03:01
À : Haddadi Manuel
Cc : [hidden email]
Objet : Re: Debug job execution from savepoint
 
Similarly, the AbstractOperatorRestoreTestBase shows how to start en embedded cluster and submit a job from a savepoint.
see
* #restoreJob
* #createJobGraph
* MiniClusterResource

On 22.06.2018 13:53, Tzu-Li (Gordon) Tai wrote:
Hi,

The tests in Flink uses a `AbstractStreamOperatorTestHarness` that allows wrapping an operator, input elements into the operator, getting the outputs, and also snapshotting / restoring operator state.
I’m not sure of your specific case, but in general that test harness utility can be used to test the operator’s output and state update logic for a given set of input records.

Cheers,
Gordon

On 22 June 2018 at 11:23:28 AM, Haddadi Manuel ([hidden email]) wrote:

Hi Fabian,


thanks for your response. Here is the Jira issue : https://issues.apache.org/jira/browse/FLINK-9644


Would you have any devlopper tips on how testing my intializeState methods to validate that my savepoints are corrects ?


Thank you again. Regards,


Manuel

De : Fabian Hueske [hidden email]
Envoyé : jeudi 21 juin 2018 10:33:10
À : Haddadi Manuel
Cc : [hidden email]
Objet : Re: Debug job execution from savepoint
 
Hi Manuel,

I had a look and couldn't find a way to do it.
However, this sounds like a very useful feature to me.

Would you mind creating a Jira issue [1] for that?

Thanks, Fabian


2018-06-18 16:23 GMT+02:00 Haddadi Manuel <[hidden email]>:

Hi all,


I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE.


Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ?


Thanks,


Manuel
Free forum by Nabble Edit this page