FLINK YARN SHIP from S3 Directory

[Vijayendra Yadav]

Hi Team,

Is there any feature to be able to ship directory to containers from s3 Directory instead of local.

-yt,--yarnship <arg>                 Ship files in the specified directory
                                          (t for transfer)

[Tzu-Li (Gordon) Tai]

Hi,

As far as I can tell from a recent change [1], this seems to be possible now starting from Flink 1.11.x. Have you already tried this with the latest Flink version?
Also including Klou in this email, who might be able to confirm this.

Cheers,
Gordon

[1] https://issues.apache.org/jira/browse/FLINK-17632

On Fri, Sep 4, 2020 at 2:02 AM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there any feature to be able to ship directory to containers from s3 Directory instead of local.

-yt,--yarnship <arg>                 Ship files in the specified directory
                                          (t for transfer)

[Vijayendra Yadav]

Thank You Gordon. I am still on 1.10.0. I will try 1.11.0 When I get there.

Regards,

Vijay

On Fri, Sep 4, 2020 at 12:52 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

As far as I can tell from a recent change [1], this seems to be possible now starting from Flink 1.11.x. Have you already tried this with the latest Flink version?
Also including Klou in this email, who might be able to confirm this.

Cheers,
Gordon

[1] https://issues.apache.org/jira/browse/FLINK-17632

On Fri, Sep 4, 2020 at 2:02 AM Vijayendra Yadav <[hidden email]> wrote:

Hi Team,

Is there any feature to be able to ship directory to containers from s3 Directory instead of local.

-yt,--yarnship <arg>                 Ship files in the specified directory
                                          (t for transfer)