Flink Cluster is savepointing jobmanager instead of external filesystem

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Cluster is savepointing jobmanager instead of external filesystem

Jason Brelloch
Hey all,

We are running into an issue where the savepoint command is saving to the jobmanager instead of the filesystem location specified in the flink-conf.yaml.  It saves to the filesystem when we are running in a local standalone flink, but when we deploy the job to our flink cluster it saves to the jobmanager.  The flink-conf is nearly identical in both places.  Here are the relevant savepoint parameters.

savepoints.state.backend: filesystem
savepoints.state.backend.fs.dir: gs://bettercloud-non-prod/flink-savepoints

Is there another configuration we have to set that is specific to clusters to make this work?  Or can anyone think of a reason this would occur?

--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster is savepointing jobmanager instead of external filesystem

Aljoscha Krettek
Hi,
are you running this standalone or on YARN?

I'm also directly looping in Ufuk because he knows that stuff best. Maybe he has an idea what could be going on.

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 18:39 Jason Brelloch <[hidden email]> wrote:
Hey all,

We are running into an issue where the savepoint command is saving to the jobmanager instead of the filesystem location specified in the flink-conf.yaml.  It saves to the filesystem when we are running in a local standalone flink, but when we deploy the job to our flink cluster it saves to the jobmanager.  The flink-conf is nearly identical in both places.  Here are the relevant savepoint parameters.

savepoints.state.backend: filesystem
savepoints.state.backend.fs.dir: gs://bettercloud-non-prod/flink-savepoints

Is there another configuration we have to set that is specific to clusters to make this work?  Or can anyone think of a reason this would occur?

--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster is savepointing jobmanager instead of external filesystem

Jason Brelloch
It is a standalone cluster.

On Fri, Oct 14, 2016 at 1:02 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
are you running this standalone or on YARN?

I'm also directly looping in Ufuk because he knows that stuff best. Maybe he has an idea what could be going on.

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 18:39 Jason Brelloch <[hidden email]> wrote:
Hey all,

We are running into an issue where the savepoint command is saving to the jobmanager instead of the filesystem location specified in the flink-conf.yaml.  It saves to the filesystem when we are running in a local standalone flink, but when we deploy the job to our flink cluster it saves to the jobmanager.  The flink-conf is nearly identical in both places.  Here are the relevant savepoint parameters.

savepoints.state.backend: filesystem
savepoints.state.backend.fs.dir: gs://bettercloud-non-prod/flink-savepoints

Is there another configuration we have to set that is specific to clusters to make this work?  Or can anyone think of a reason this would occur?

--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox



--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster is savepointing jobmanager instead of external filesystem

Aljoscha Krettek
Which version of Flink are you using? In the 1.2 SNAPSHOT version the configuration for save points is changed: https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/savepoints.html#configuration

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 19:03 Jason Brelloch <[hidden email]> wrote:
It is a standalone cluster.

On Fri, Oct 14, 2016 at 1:02 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
are you running this standalone or on YARN?

I'm also directly looping in Ufuk because he knows that stuff best. Maybe he has an idea what could be going on.

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 18:39 Jason Brelloch <[hidden email]> wrote:
Hey all,

We are running into an issue where the savepoint command is saving to the jobmanager instead of the filesystem location specified in the flink-conf.yaml.  It saves to the filesystem when we are running in a local standalone flink, but when we deploy the job to our flink cluster it saves to the jobmanager.  The flink-conf is nearly identical in both places.  Here are the relevant savepoint parameters.

savepoints.state.backend: filesystem
savepoints.state.backend.fs.dir: gs://bettercloud-non-prod/flink-savepoints

Is there another configuration we have to set that is specific to clusters to make this work?  Or can anyone think of a reason this would occur?

--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox



--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster is savepointing jobmanager instead of external filesystem

Jason Brelloch
Sorry, I should have mentioned that.  We are still running 1.0.1 version.

On Sun, Oct 16, 2016 at 5:08 AM, Aljoscha Krettek <[hidden email]> wrote:
Which version of Flink are you using? In the 1.2 SNAPSHOT version the configuration for save points is changed: https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/savepoints.html#configuration

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 19:03 Jason Brelloch <[hidden email]> wrote:
It is a standalone cluster.

On Fri, Oct 14, 2016 at 1:02 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,
are you running this standalone or on YARN?

I'm also directly looping in Ufuk because he knows that stuff best. Maybe he has an idea what could be going on.

Cheers,
Aljoscha

On Fri, 14 Oct 2016 at 18:39 Jason Brelloch <[hidden email]> wrote:
Hey all,

We are running into an issue where the savepoint command is saving to the jobmanager instead of the filesystem location specified in the flink-conf.yaml.  It saves to the filesystem when we are running in a local standalone flink, but when we deploy the job to our flink cluster it saves to the jobmanager.  The flink-conf is nearly identical in both places.  Here are the relevant savepoint parameters.

savepoints.state.backend: filesystem
savepoints.state.backend.fs.dir: gs://bettercloud-non-prod/flink-savepoints

Is there another configuration we have to set that is specific to clusters to make this work?  Or can anyone think of a reason this would occur?

--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox



--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox



--
Jason Brelloch | Product Developer
3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 
Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox