Custom S3 endpoint

[Martin, Nick-2]

I’m working on Flink 1.7.0 and I’m trying to use the built in S3 libraries like readFile(‘s3://bucket/object’) or StreamingFileSink. My storage provider is not AWS, but they implement the same API. So I need to point the S3 client to a different address. The Hadoop documentation shows that there are options in core-site to set that up. The problem is, I can’t seem to get the right dependencies together to use the S3 filesystem. As far as I can tell, the pre-built Hadoop/Presto jars don’t use core-site.xml, and the instructions for manual setup given here (https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup) list a set of dependencies that seems to be completely wrong.

 

How can use the S3 sources/sinks with a custom http endpoint?

 

 

 

Nick Martin

 

---

Notice: This e-mail is intended solely for use of the individual or entity to which it is addressed and may contain information that is proprietary, privileged and/or exempt from disclosure under applicable law. If the reader is not the intended recipient or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. This communication may also contain data subject to U.S. export laws. If so, data subject to the International Traffic in Arms Regulation cannot be disseminated, distributed, transferred, or copied, whether incorporated or in its original form, to foreign nationals residing in the U.S. or abroad, absent the express prior approval of the U.S. Department of State. Data subject to the Export Administration Act may not be disseminated, distributed, transferred or copied contrary to U. S. Department of Commerce regulations. If you have received this communication in error, please notify the sender by reply e-mail and destroy the e-mail message and any physical copies made of the communication.
 Thank you. 
*********************

[Paul Lam]

Hi Nick,

What version of Hadoop are you using? AFAIK, you must use Hadoop 2.7+ to support custom s3 endpoint, or the `fs.s3a.endpoint` property in core-site.xml would be ignored.

Best,

Paul Lam

在 2018年12月19日，06:40，Martin, Nick <[hidden email]> 写道：

I’m working on Flink 1.7.0 and I’m trying to use the built in S3 libraries like readFile(‘<a href="s3://bucket/object%E2%80%99" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">s3://bucket/object’) or StreamingFileSink. My storage provider is not AWS, but they implement the same API. So I need to point the S3 client to a different address. The Hadoop documentation shows that there are options in core-site to set that up. The problem is, I can’t seem to get the right dependencies together to use the S3 filesystem. As far as I can tell, the pre-built Hadoop/Presto jars don’t use core-site.xml, and the instructions for manual setup given here (https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup) list a set of dependencies that seems to be completely wrong. 

 

How can use the S3 sources/sinks with a custom http endpoint?

 

 

 

Nick Martin

 

---

Notice: This e-mail is intended solely for use of the individual or entity to which it is addressed and may contain information that is proprietary, privileged and/or exempt from disclosure under applicable law. If the reader is not the intended recipient or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. This communication may also contain data subject to U.S. export laws. If so, data subject to the International Traffic in Arms Regulation cannot be disseminated, distributed, transferred, or copied, whether incorporated or in its original form, to foreign nationals residing in the U.S. or abroad, absent the express prior approval of the U.S. Department of State. Data subject to the Export Administration Act may not be disseminated, distributed, transferred or copied contrary to U. S. Department of Commerce regulations. If you have received this communication in error, please notify the sender by reply e-mail and destroy the e-mail message and any physical copies made of the communication.
 Thank you. 
*********************

[Martin, Nick-2]

Yeah, I figured that part out. I’ve tried to make it work with 2.7 and 2.8, and it looks like the prebuilt jars have actually moved to Hadoop 3

 

From: Paul Lam [mailto:[hidden email]]
Sent: Tuesday, December 18, 2018 7:08 PM
To: Martin, Nick <[hidden email]>
Cc: [hidden email]
Subject: EXT :Re: Custom S3 endpoint

 

Hi Nick,

 

What version of Hadoop are you using? AFAIK, you must use Hadoop 2.7+ to support custom s3 endpoint, or the `fs.s3a.endpoint` property in core-site.xml would be ignored.

 

Best,

Paul Lam

在 2018年12月19日，06:40，Martin, Nick <[hidden email]> 写道：

 

I’m working on Flink 1.7.0 and I’m trying to use the built in S3 libraries like readFile(‘<a href="s3://bucket/object%E2%80%99">s3://bucket/object’) or StreamingFileSink. My storage provider is not AWS, but they implement the same API. So I need to point the S3 client to a different address. The Hadoop documentation shows that there are options in core-site to set that up. The problem is, I can’t seem to get the right dependencies together to use the S3 filesystem. As far as I can tell, the pre-built Hadoop/Presto jars don’t use core-site.xml, and the instructions for manual setup given here (https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup) list a set of dependencies that seems to be completely wrong. 

 

How can use the S3 sources/sinks with a custom http endpoint?

 

 

 

Nick Martin

 

 

---

Notice: This e-mail is intended solely for use of the individual or entity to which it is addressed and may contain information that is proprietary, privileged and/or exempt from disclosure under applicable law. If the reader is not the intended recipient or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. This communication may also contain data subject to U.S. export laws. If so, data subject to the International Traffic in Arms Regulation cannot be disseminated, distributed, transferred, or copied, whether incorporated or in its original form, to foreign nationals residing in the U.S. or abroad, absent the express prior approval of the U.S. Department of State. Data subject to the Export Administration Act may not be disseminated, distributed, transferred or copied contrary to U. S. Department of Commerce regulations. If you have received this communication in error, please notify the sender by reply e-mail and destroy the e-mail message and any physical copies made of the communication.
 Thank you. 
*********************

 

 

---

Notice: This e-mail is intended solely for use of the individual or entity to which it is addressed and may contain information that is proprietary, privileged and/or exempt from disclosure under applicable law. If the reader is not the intended recipient or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. This communication may also contain data subject to U.S. export laws. If so, data subject to the International Traffic in Arms Regulation cannot be disseminated, distributed, transferred, or copied, whether incorporated or in its original form, to foreign nationals residing in the U.S. or abroad, absent the express prior approval of the U.S. Department of State. Data subject to the Export Administration Act may not be disseminated, distributed, transferred or copied contrary to U. S. Department of Commerce regulations. If you have received this communication in error, please notify the sender by reply e-mail and destroy the e-mail message and any physical copies made of the communication.
 Thank you. 
*********************

---

Notice: This e-mail is intended solely for use of the individual or entity to which it is addressed and may contain information that is proprietary, privileged and/or exempt from disclosure under applicable law. If the reader is not the intended recipient or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. This communication may also contain data subject to U.S. export laws. If so, data subject to the International Traffic in Arms Regulation cannot be disseminated, distributed, transferred, or copied, whether incorporated or in its original form, to foreign nationals residing in the U.S. or abroad, absent the express prior approval of the U.S. Department of State. Data subject to the Export Administration Act may not be disseminated, distributed, transferred or copied contrary to U. S. Department of Commerce regulations. If you have received this communication in error, please notify the sender by reply e-mail and destroy the e-mail message and any physical copies made of the communication.
 Thank you. 
*********************