(DEPRECATED) Apache Flink User Mailing List archive.

Questions regarding to Flink 1.5.0 REST API change

Classic

List

Threaded

12 messages Options

Siew Wai Yow

Questions regarding to Flink 1.5.0 REST API change

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?
1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
1. previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Esteban Serrano

Re: Questions regarding to Flink 1.5.0 REST API change

For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Chesnay Schepler

Re: Questions regarding to Flink 1.5.0 REST API change

1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{



        "target-directory" : {

      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}

3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:

For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.

The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".

For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Siew Wai Yow

Re: Questions regarding to Flink 1.5.0 REST API change

Thank you [hidden email] and [hidden email]!

From: Chesnay Schepler <[hidden email]>
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{



"target-directory" : {

      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}

3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:

For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Siew Wai Yow

Re: Questions regarding to Flink 1.5.0 REST API change

In reply to this post by Chesnay Schepler

Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.

From: Chesnay Schepler <[hidden email]>
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{



"target-directory" : {

      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}

3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:

For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Chesnay Schepler

Re: Questions regarding to Flink 1.5.0 REST API change

I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:

Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.

The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".

For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Siew Wai Yow

Re: Questions regarding to Flink 1.5.0 REST API change

Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow

From: Chesnay Schepler <[hidden email]>
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

gerryzhou

Re: Questions regarding to Flink 1.5.0 REST API change

Hi Yow,

I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint.

Best, Sihua

On 06/21/2018 14:07，[hidden email] wrote：

Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow
From: Chesnay Schepler <[hidden email]>
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Chesnay Schepler

Re: Questions regarding to Flink 1.5.0 REST API change

In reply to this post by Siew Wai Yow

That's quite weird that it tries to us the local file-system. Maybe it derives the FS from state.backend.fs.checckpointdir, but uses it for savepoints.dir?

What happens if you set state.backend.fs.checkpointdir also to HDFS?

On 21.06.2018 08:07, Siew Wai Yow wrote:

Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow
From: Chesnay Schepler [hidden email]
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.

The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".

For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

gerryzhou

Re: Questions regarding to Flink 1.5.0 REST API change

I just create a JIRA for this: https://issues.apache.org/jira/browse/FLINK-9633

On 06/21/2018 15:10，[hidden email] wrote：

That's quite weird that it tries to us the local file-system. Maybe it derives the FS from state.backend.fs.checckpointdir, but uses it for savepoints.dir?

What happens if you set state.backend.fs.checkpointdir also to HDFS?

On 21.06.2018 08:07, Siew Wai Yow wrote:
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow
From: Chesnay Schepler [hidden email]
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.

The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".

For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Chesnay Schepler

Re: Questions regarding to Flink 1.5.0 REST API change

In reply to this post by gerryzhou

If you could open a JIRA this would be great.

On 21.06.2018 09:07, sihua zhou wrote:

Hi Yow,

I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint.

Best, Sihua

On 06/21/2018 14:07，[hidden email] wrote：
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow
From: Chesnay Schepler <[hidden email]>
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.

The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".

For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow

Siew Wai Yow

Re: Questions regarding to Flink 1.5.0 REST API change

Thanks @SihuaZhou, you are right that this is a bug. Just check the source code too. [hidden email]Tested with both checkpoint and savepoint at same File system and it is working as expected.

Thanks guys!

-Yow

From: Chesnay Schepler <[hidden email]>
Sent: Thursday, June 21, 2018 3:36 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

If you could open a JIRA this would be great.

On 21.06.2018 09:07, sihua zhou wrote:

Hi Yow,

I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint.

Best, Sihua

On 06/21/2018 14:07，[hidden email] wrote：
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more

flink-conf.yaml:

--------------------------------

state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints

-Yow
From: Chesnay Schepler <[hidden email]>
Sent: Wednesday, June 20, 2018 3:15 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you.
From: Chesnay Schepler [hidden email]
Sent: Tuesday, June 19, 2018 11:55 PM
To: [hidden email]
Subject: Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
{"target-directory" : {
      "type" : "string"
    },
    "cancel-job" : {
      "type" : "boolean"
    }
}
3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[hidden email]> wrote:

Hi,

Regarding to Flink 1.5.0 REST API breaking change,

The REST API to cancel a job was changed.
The REST API to cancel a job with savepoint was changed.

I have few dump questions,

Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward?

http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??

Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?

previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.

Regards,

Yow