(DEPRECATED) Apache Flink User Mailing List archive.

Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Fabian Wollert-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

Fabian Wollert-2
Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem backend. I configured the following:

state.backend=filesystem
state.backend.fs.checkpointdir=s3:///mybucket/
state.checkpoints.dir=s3:///mybucket/
state.checkpoints.num-retained=3

I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.

I get now though the following error message:

Caused by: java.lang.NullPointerException: null uri host.
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
    at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)

I tried to dig deeper into the source code, but struggled to find 
  • what is meant with this URI
  • where to configure it
Can anybody give some advice how to set up the S3 Backend with the new shaded lib jar?

Thanks in advance
--

Fabian Wollert
Zalando SE

Tamara-Danz-Straße 1
10243 Berlin
Fax: +49 (0)30 2759 46 93
Notifications of major holdings (Sec. 33, 38, 39 WpHG):  +49 (0)30 2000889349

Management Board:
Robert Gentz, David Schneider, Rubin Ritter

Chairman of the Supervisory Board:
Lothar Lanz

Person responsible for providing the contents of Zalando SE acc. to Art. 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
VAT registration number: DE 260543043
phoenixjiangnan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

phoenixjiangnan
Did you run Flink on AWS EMR or somewhere else? Have you read and followed instructions on https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws  ?



On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <[hidden email]> wrote:
Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem backend. I configured the following:

state.backend=filesystem
state.backend.fs.checkpointdir=s3:///mybucket/
state.checkpoints.dir=s3:///mybucket/
state.checkpoints.num-retained=3

I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.

I get now though the following error message:

Caused by: java.lang.NullPointerException: null uri host.
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
    at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)

I tried to dig deeper into the source code, but struggled to find 
  • what is meant with this URI
  • where to configure it
Can anybody give some advice how to set up the S3 Backend with the new shaded lib jar?

Thanks in advance
--

Fabian Wollert
Zalando SE

Fax: +49 (0)30 2759 46 93
Notifications of major holdings (Sec. 33, 38, 39 WpHG):  +49 (0)30 2000889349

Management Board:
Robert Gentz, David Schneider, Rubin Ritter

Chairman of the Supervisory Board:
Lothar Lanz

Person responsible for providing the contents of Zalando SE acc. to Art. 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
VAT registration number: DE 260543043
Fabian Wollert-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

Fabian Wollert-2
I'm running it in docker on EC2, cant use EMR ... yes i followed those instructions.

Cheers

--

Fabian Wollert
Zalando SE


Am Do., 31. Mai 2018 um 03:07 Uhr schrieb Bowen Li <[hidden email]>:
Did you run Flink on AWS EMR or somewhere else? Have you read and followed instructions on https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws  ?



On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <[hidden email]> wrote:
Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem backend. I configured the following:

state.backend=filesystem
state.backend.fs.checkpointdir=s3:///mybucket/
state.checkpoints.dir=s3:///mybucket/
state.checkpoints.num-retained=3

I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.

I get now though the following error message:

Caused by: java.lang.NullPointerException: null uri host.
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
    at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)

I tried to dig deeper into the source code, but struggled to find 
  • what is meant with this URI
  • where to configure it
Can anybody give some advice how to set up the S3 Backend with the new shaded lib jar?

Thanks in advance
--

Fabian Wollert
Zalando SE

Fax: +49 (0)30 2759 46 93
Notifications of major holdings (Sec. 33, 38, 39 WpHG):  +49 (0)30 2000889349

Management Board:
Robert Gentz, David Schneider, Rubin Ritter

Chairman of the Supervisory Board:
Lothar Lanz

Person responsible for providing the contents of Zalando SE acc. to Art. 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
VAT registration number: DE 260543043
Fabian Wollert-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

Fabian Wollert-2

i solved it by myself, with the help of some debugging. i used s3:///mybucket/ but it needs to be s3://mybucket/some_folder ... 2 slashes, and also a folder needs to be specified ...
--

Fabian Wollert
Zalando SE
Location: ZMAP



Am Do., 31. Mai 2018 um 09:31 Uhr schrieb Fabian Wollert <[hidden email]>:
I'm running it in docker on EC2, cant use EMR ... yes i followed those instructions.

Cheers

--

Fabian Wollert
Zalando SE


Am Do., 31. Mai 2018 um 03:07 Uhr schrieb Bowen Li <[hidden email]>:
Did you run Flink on AWS EMR or somewhere else? Have you read and followed instructions on https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws  ?



On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <[hidden email]> wrote:
Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem backend. I configured the following:

state.backend=filesystem
state.backend.fs.checkpointdir=s3:///mybucket/
state.checkpoints.dir=s3:///mybucket/
state.checkpoints.num-retained=3

I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.

I get now though the following error message:

Caused by: java.lang.NullPointerException: null uri host.
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
    at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
    at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)

I tried to dig deeper into the source code, but struggled to find 
  • what is meant with this URI
  • where to configure it
Can anybody give some advice how to set up the S3 Backend with the new shaded lib jar?

Thanks in advance
--

Fabian Wollert
Zalando SE

Fax: +49 (0)30 2759 46 93
Notifications of major holdings (Sec. 33, 38, 39 WpHG):  +49 (0)30 2000889349

Management Board:
Robert Gentz, David Schneider, Rubin Ritter

Chairman of the Supervisory Board:
Lothar Lanz

Person responsible for providing the contents of Zalando SE acc. to Art. 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
VAT registration number: DE 260543043
Free forum by Nabble Edit this page