Kinesis Connectors - With Temporary Credentials

[sreenath kodedala]

>
> Hi,
>
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect.
>
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector?
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.
>
> Thank you
> -Sree

[Tzu-Li (Gordon) Tai]

Hi Sree,

Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?

If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.

To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.

Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.

Cheers,

Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html

On 12 January 2018 at 7:25:58 AM, sreenath kodedala ([hidden email]) wrote:

>
> Hi,
>
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect.
>
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector?
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.
>
> Thank you
> -Sree

[sreenath kodedala]

No, they are not but we can definitely look into that.

If no, is there a workaround to implement or customize AWS Utils?

Thank you

On Jan 11, 2018, at 6:41 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Sree,

Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?

If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.

To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.

Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.

Cheers,

Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html

On 12 January 2018 at 7:25:58 AM, sreenath kodedala ([hidden email]) wrote:

>  
> Hi, 
>  
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect. 
>  
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector? 
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.  
>  
> Thank you 
> -Sree

[Tzu-Li (Gordon) Tai]

Ah, I see. Temporary Credentials are delegated through the AWS Security Token Service through the AssumeRole API.

Sorry, I wasn’t knowledgable of the Temporary Credentials feature before.

Seems like we should add support for the STSAssumeRoleSessionCredentialsProvider [1]. And yes, your observation is correct that I think this would be a matter of extending the AWSUtil class.

I’ve filed a JIRA for the issue: FLINK-8417 [2]. Would you like to contribute this feature? That would be of great help and I think it’ll be a useful addition. If yes, feel free to ping me for any questions you may have.

Cheers,

Gordon

[1] https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html

[2] https://issues.apache.org/jira/browse/FLINK-8417

On 12 January 2018 at 7:46:10 AM, sreenath kodedala ([hidden email]) wrote:

No, they are not but we can definitely look into that.

If no, is there a workaround to implement or customize AWS Utils?

Thank you

On Jan 11, 2018, at 6:41 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Sree,

Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?

If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.

To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.

Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.

Cheers,

Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html

On 12 January 2018 at 7:25:58 AM, sreenath kodedala ([hidden email]) wrote:

>  
> Hi, 
>  
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect. 
>  
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector? 
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.  
>  
> Thank you 
> -Sree