ClusterClientFactory selection

[M Singh]

Hi:

I wanted to find out which parameter/configuration allows flink cli pick up the appropriate cluster client factory (especially in the yarn mode).

Thanks

[Yang Wang]

Hi M Singh,

The Flink CLI picks up the correct ClusterClientFactory via java SPI. You

could check YarnClusterClientFactory#isCompatibleWith for how it is activated.

The cli option / configuration is "-e/--executor" or execution.target (e.g. yarn-per-job).

Best,
Yang

M Singh <[hidden email]> 于2020年5月26日周二 下午6:45写道：

Hi:

I wanted to find out which parameter/configuration allows flink cli pick up the appropriate cluster client factory (especially in the yarn mode).

Thanks

[Kostas Kloudas-2]

Hi Singh,

The only thing to add to what Yang said is that the "execution.target"
configuration option (in the config file) is also used for the same
purpose from the execution environments.

Cheers,
Kostas

On Wed, May 27, 2020 at 4:49 AM Yang Wang <[hidden email]> wrote:

>
> Hi M Singh,
>
> The Flink CLI picks up the correct ClusterClientFactory via java SPI. You
> could check YarnClusterClientFactory#isCompatibleWith for how it is activated.
> The cli option / configuration is "-e/--executor" or execution.target (e.g. yarn-per-job).
>
>
> Best,
> Yang
>
> M Singh <[hidden email]> 于2020年5月26日周二 下午6:45写道：
>>
>> Hi:
>>
>> I wanted to find out which parameter/configuration allows flink cli pick up the appropriate cluster client factory (especially in the yarn mode).
>>
>> Thanks

[M Singh]

HI Kostas/Yang/Lake:

I am looking at aws emr and did not see the execution.target in the flink-conf.yaml file under flink/conf directory. 

Is it defined in another place ?  

I also did search in the current flink source code and did find mention of it in the md files but not in any property file or the flink-yarn sub module.  

Please let me know if I am missing anything.

Thanks

On Wednesday, May 27, 2020, 03:51:28 AM EDT, Kostas Kloudas <[hidden email]> wrote:

Hi Singh,

The only thing to add to what Yang said is that the "execution.target"
configuration option (in the config file) is also used for the same
purpose from the execution environments.

Cheers,
Kostas

On Wed, May 27, 2020 at 4:49 AM Yang Wang <[hidden email]> wrote:

>
> Hi M Singh,
>
> The Flink CLI picks up the correct ClusterClientFactory via java SPI. You
> could check YarnClusterClientFactory#isCompatibleWith for how it is activated.
> The cli option / configuration is "-e/--executor" or execution.target (e.g. yarn-per-job).
>
>
> Best,
> Yang
>
> M Singh <[hidden email]> 于2020年5月26日周二 下午6:45写道：
>>
>> Hi:
>>
>> I wanted to find out which parameter/configuration allows flink cli pick up the appropriate cluster client factory (especially in the yarn mode).
>>
>> Thanks

[Yang Wang]

You could find more information about deployment target here[1]. As you mentioned,

it is not defined in the flink-conf.yaml by default.

For the code, it is defined in flink-core/DeploymentOptions.

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/cli.html#deployment-targets

Best,
Yang

M Singh <[hidden email]> 于2020年5月28日周四 下午10:34写道：

HI Kostas/Yang/Lake:

I am looking at aws emr and did not see the execution.target in the flink-conf.yaml file under flink/conf directory. 

Is it defined in another place ?  

I also did search in the current flink source code and did find mention of it in the md files but not in any property file or the flink-yarn sub module.  

Please let me know if I am missing anything.

Thanks

On Wednesday, May 27, 2020, 03:51:28 AM EDT, Kostas Kloudas <[hidden email]> wrote:

Hi Singh,

The only thing to add to what Yang said is that the "execution.target"
configuration option (in the config file) is also used for the same
purpose from the execution environments.

Cheers,
Kostas

On Wed, May 27, 2020 at 4:49 AM Yang Wang <[hidden email]> wrote:

>
> Hi M Singh,
>
> The Flink CLI picks up the correct ClusterClientFactory via java SPI. You
> could check YarnClusterClientFactory#isCompatibleWith for how it is activated.
> The cli option / configuration is "-e/--executor" or execution.target (e.g. yarn-per-job).
>
>
> Best,
> Yang
>
> M Singh <[hidden email]> 于2020年5月26日周二 下午6:45写道：
>>
>> Hi:
>>
>> I wanted to find out which parameter/configuration allows flink cli pick up the appropriate cluster client factory (especially in the yarn mode).
>>
>> Thanks