Using sensitive configuration/credentials

[Matt Moore]

I'm wondering what the best practice is for using secrets in a Flink program, and I can't find any info in the docs or posted anywhere else. 

I need to store an access token to one of my APIs for flink to use to dump results into, and right now I'm passing it through as a configuration parameter, but that doesn't seem like the most secure thing to do and the value shows up in the Flink Dashboard under Configuration which is less than ideal. 

Has anyone else dealt with a situation like this?

Thanks,

[vino yang]

Hi Matt,

Flink is currently enhancing its security, such as the current data transmission can be configured with SSL mode[1]. 

However, some problems involving configuration and web ui display do exist, and they are still displayed in plain text. 

I think a temporary way to do this is to keep your secret configuration in encrypted form elsewhere, such as Zookeeper or RDBMS, and then dynamically read it into the job in a UDF (in the open method).

https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html

Thanks, vino.

Matt Moore <[hidden email]> 于2018年8月9日周四 上午1:54写道：

I'm wondering what the best practice is for using secrets in a Flink program, and I can't find any info in the docs or posted anywhere else. 

I need to store an access token to one of my APIs for flink to use to dump results into, and right now I'm passing it through as a configuration parameter, but that doesn't seem like the most secure thing to do and the value shows up in the Flink Dashboard under Configuration which is less than ideal. 

Has anyone else dealt with a situation like this?

Thanks,

[Chesnay Schepler]

If you change the name of your configuration key ti include "secret" or "password" it should be hidden from the logs and UI.

On 09.08.2018 04:28, vino yang wrote:

Hi Matt,

Flink is currently enhancing its security, such as the current data transmission can be configured with SSL mode[1]. 

However, some problems involving configuration and web ui display do exist, and they are still displayed in plain text. 

I think a temporary way to do this is to keep your secret configuration in encrypted form elsewhere, such as Zookeeper or RDBMS, and then dynamically read it into the job in a UDF (in the open method).

https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html

Thanks, vino.

Matt Moore <[hidden email]> 于2018年8月9日周四 上午1:54写道：

I'm wondering what the best practice is for using secrets in a Flink program, and I can't find any info in the docs or posted anywhere else. 

I need to store an access token to one of my APIs for flink to use to dump results into, and right now I'm passing it through as a configuration parameter, but that doesn't seem like the most secure thing to do and the value shows up in the Flink Dashboard under Configuration which is less than ideal. 

Has anyone else dealt with a situation like this?

Thanks,

[vino yang]

Hi Chesnay,

Oh, I did not know this feature. Any more description in Flink official documentation?

Thanks, vino.

Chesnay Schepler <[hidden email]> 于2018年8月9日周四 下午4:29写道：

If you change the name of your configuration key ti include "secret" or "password" it should be hidden from the logs and UI.

On 09.08.2018 04:28, vino yang wrote:

Hi Matt,

Flink is currently enhancing its security, such as the current data transmission can be configured with SSL mode[1]. 

However, some problems involving configuration and web ui display do exist, and they are still displayed in plain text. 

I think a temporary way to do this is to keep your secret configuration in encrypted form elsewhere, such as Zookeeper or RDBMS, and then dynamically read it into the job in a UDF (in the open method).

https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html

Thanks, vino.

Matt Moore <[hidden email]> 于2018年8月9日周四 上午1:54写道：

I'm wondering what the best practice is for using secrets in a Flink program, and I can't find any info in the docs or posted anywhere else. 

I need to store an access token to one of my APIs for flink to use to dump results into, and right now I'm passing it through as a configuration parameter, but that doesn't seem like the most secure thing to do and the value shows up in the Flink Dashboard under Configuration which is less than ideal. 

Has anyone else dealt with a situation like this?

Thanks,