Ask for reason for choice of S3 plugins

[B.Zhou]

Hi,

 

In this document https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins, it mentioned that

• Presto is the recommended file system for checkpointing to S3.

Is there a reason for that? Is there some bottleneck for s3 hadoop plugin that can’t support checkpoint storage well?

 

And if I have the s3:// scheme with both plugins loaded, is there a class loading order or just random for accessing S3? Which plugin will take charge?

 

Best Regards,

Brian

 

[David Anderson-2]

If you are using both the Hadoop S3 and Presto S3 filesystems, you should use s3p:// and s3a:// to distinguish between the two.

Presto is recommended for checkpointing because the Hadoop implementation has very high latency when creating files, and because it hits request rate limits very quickly. The Hadoop S3 filesystem tries to imitate a normal filesystem on top of S3:

 - before writing a key it checks if the "parent directory" exists by checking for a key with the prefix up to the last "/"
 - it creates empty marker files to mark the existence of such a parent directory
 - all these existence requests are S3 HEAD requests, which have very low request rate limits

On Fri, Mar 27, 2020 at 7:10 AM <[hidden email]> wrote:

Hi,

 

In this document https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins, it mentioned that

• Presto is the recommended file system for checkpointing to S3.

Is there a reason for that? Is there some bottleneck for s3 hadoop plugin that can’t support checkpoint storage well?

 

And if I have the s3:// scheme with both plugins loaded, is there a class loading order or just random for accessing S3? Which plugin will take charge?

 

Best Regards,

Brian