Detached execution API

[nragon]

It would be nice to let users deploy detached jobs through api.
For instance>

StreamExecutionEnviroment

  public JobExecutionResult execute() throws Exception {
    return execute(DEFAULT_JOB_NAME, false);
  }
Which keep backward compatibility

  public abstract JobExecutionResult execute(String jobName, boolean detached) throws Exception;

RemoteStreamEnvironment

@Override
  public JobExecutionResult execute(String jobName, boolean detached) throws ProgramInvocationException {
    StreamGraph streamGraph = getStreamGraph();
    streamGraph.setJobName(jobName);
    transformations.clear();
    return executeRemotely(streamGraph, jarFiles, detached);
  }

---------------------

protected JobExecutionResult executeRemotely(StreamGraph streamGraph, List<URL> jarFiles, boolean detached) throws ProgramInvocationException {
.....
ClusterClient client;
    try {
      client = new StandaloneClusterClient(configuration);
      client.setDetached(detached);
      client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
    }
....
}

Just an idea. I'm using the previous approach and it works fine.

Thanks

[Aljoscha Krettek]

Hi,

I think you might find those two issues interesting:

- https://issues.apache.org/jira/browse/FLINK-2313: Change Streaming Driver Execution Model
- https://issues.apache.org/jira/browse/FLINK-4272: Create a JobClient for job control and monitoring

Best,
Aljoscha

> On 20. Jul 2017, at 17:51, nragon <[hidden email]> wrote:
>
> It would be nice to let users deploy detached jobs through api.
> For instance>
>
> *StreamExecutionEnviroment*
>
>  public JobExecutionResult execute() throws Exception {
>    return execute(DEFAULT_JOB_NAME, false);
>  }
> Which keep backward compatibility
>
>  public abstract JobExecutionResult execute(String jobName, boolean
> detached) throws Exception;
>
> *RemoteStreamEnvironment*
>
> @Override
>  public JobExecutionResult execute(String jobName, boolean detached) throws
> ProgramInvocationException {
>    StreamGraph streamGraph = getStreamGraph();
>    streamGraph.setJobName(jobName);
>    transformations.clear();
>    return executeRemotely(streamGraph, jarFiles, detached);
>  }
>
> ---------------------
>
> protected JobExecutionResult executeRemotely(StreamGraph streamGraph,
> List<URL> jarFiles, boolean detached) throws ProgramInvocationException {
> .....
> ClusterClient client;
>    try {
>      client = new StandaloneClusterClient(configuration);
>      client.setDetached(detached);
>
> client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
>    }
> ....
> }
>
> Just an idea. I'm using the previous approach and it works fine.
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Detached-execution-API-tp14366.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

Yes, something like that.
Although, this would be an MVP for this purpose which has minimal impacts on next releases. Moreover, I think clients are too specific to be included as API because each user will have their own way to implement a stop, start, or whatever (of course a base client is welcome :)). For instance, we have implemented one, using REST API + async(detached) job.

Also, one thing I forgot in the previous example is that abstract execute should return a JobSubmissionResult instead of JobExecutionResult.

But of course, just my point of view.