In-transit Data Encryption in EMR
Locked
In-transit Data Encryption in EMR
 
|
Hi,
Currently I am looking into configuring in-transit data encryption either using Flink SSL Setup or directly using EMR. Few Doubts: 1. Will the existing functionality provided by Amazon to configure in-transit data encrytion work for Flink as well. This is explained here: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit 2. Using Flink SSL Setup: as we know only the IP address of master node on EMR , should we pass only its ip address in the SAN list as given here ? (I think it should work as the yarn-cli command will distribute the truststore and keystore to each TM ) https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore Regards, Vinay Patil |
Re: In-transit Data Encryption in EMR
|
|
Hi Vinay, I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features. Cheers, Till On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email]> wrote: Hi, |
Re: In-transit Data Encryption in EMR
|
|
Thank you Till. Gordon can you please help. Regards, Vinay Patil On Fri, Jun 2, 2017 at 9:10 PM, Till Rohrmann [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
|
Re: In-transit Data Encryption in EMR
|
|
Hi Vinay!
I’m not entirely sure of the subdomain patterns of EMR nodes, but this should be possible.
Cheers, Gordon On 5 June 2017 at 12:56:45 PM, vinay patil ([hidden email]) wrote:
|
Re: In-transit Data Encryption in EMR
|
|
Hi Gordan,
Thank you for your response. I have done the necessary configurations by adding all the node ip's from Resource Manager , is this correct ? Also I will try to check if wildcard works as all our hostname begins with a same pattern. For ex : SAN=dns:ip-192-168.* should work , right ? Facing a weird issue when I try to submit the job using the following command: flink run -m yarn-cluster -yn 4 -ys 4 -yjm 1024 -ytm 4000 -yt deploy-keys/ testFlinkSSL.jar --configFileName conf.yaml Error is : java.lang.IllegalArgumentException: Wrong FS: hdfs://<some_ip>:8020/user/hadoop/.flink/application_1496660166576_0001/flink-dist_2.10-1.2.0.jar, expected: file:/// I see a JIRA ticket regarding the same but did not find any solution to this. Regards, Vinay Patil |
Re: In-transit Data Encryption in EMR
|
|
Hi Gordon,
The yarn session gets created when I try to run the following command: yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/ However when I try to access the Job Manager UI, it gives me exception as : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target I am able to see the Job Manager UI when I imported the CA certificate to java truststore on EMR master node : keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ? Also the keystore and truststore password should be masked in the logs which is not case. 2017-06-05 14:51:31,135 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.enabled, true 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.keystore-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.key-password, password 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore 2017-06-05 14:51:31,136 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.ssl.truststore-password, password Regards, Vinay Patil |
Re: In-transit Data Encryption in EMR
|
|
Hi Guys, I am able to setup SSL correctly, however the following command does not work correctly and results in the error I had mailed earlier Few Doubts: 1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs. 2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command: keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* Regards, Vinay Patil On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <[hidden email]> wrote: Hi Gordon, |
Re: In-transit Data Encryption in EMR
|
|
In reply to this post by Vinay Patil
Hi Guys, Can anyone please provide me solution to my queries. On Jun 8, 2017 11:30 PM, "Vinay Patil" <[hidden email]> wrote:
|
Re: In-transit Data Encryption in EMR
|
|
Hi Vinay, Apologies for the inactivity on this thread, I was occupied with some critical fixes for 1.3.1.
AFAIK, if any of the SSL configuration settings are enabled (*.ssl.enabled) and your job is running fine, then everything should be functioning.
The wildcard option only works for wildcarding subdomains. For example, SAN=*.domain.com On 9 June 2017 at 4:33:46 PM, vinay patil ([hidden email]) wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |