Why am I getting AWS access denied error for request type [DeleteObjectRequest] in S3?

[HarshithBolar]

Hi all,

 

We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.

 

Here’s a snippet of the logs for DELETE request –

 

2018-10-15 04:13:22,819 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78]

 

Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3?

 

I’d be happy to provide more information if needed.

 

Thanks,

Harshith

 

 

[Amit Jain]

Hi Harshith,

Did you enable delete permission on S3 for running machines? Are you using IAM roles or access key id and secret access key combo?

--

Thanks,

Amit

On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Hi all,

 

We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.

 

Here’s a snippet of the logs for DELETE request –

 

2018-10-15 04:13:22,819 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78]

 

Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3?

 

I’d be happy to provide more information if needed.

 

Thanks,

Harshith

 

 

[HarshithBolar]

Thanks Amit,

 

I’m now in the process of checking our IAM roles to see if the user has been given DeleteObject permission to S3. I’m guessing that’s the most likely cause for this error.

 

- Harshith

 

From: Amit Jain <[hidden email]>
Date: Monday, 15 October 2018 at 4:46 PM
To: Harshith Kumar Bolar <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: [External] Re: Why am I getting AWS access denied error for request type [DeleteObjectRequest] in S3?

 

Hi Harshith,

Did you enable delete permission on S3 for running machines? Are you using IAM roles or access key id and secret access key combo?

--

Thanks,

Amit

 

On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Hi all,

 

We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.

 

Here’s a snippet of the logs for DELETE request –

 

2018-10-15 04:13:22,819 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78]

 

Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3?

 

I’d be happy to provide more information if needed.

 

Thanks,

Harshith