Flink SSL Setup on a standalone cluster

[Vinay Patil]

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

[Timo Walther]

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

[Vinay Patil]

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[Vinay Patil]

Just an update,  I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this.

And in that I am passing the hostname which is same as provided in flink-conf.yaml 

On Thu, Mar 15, 2018 at 7:57 AM, Vinay Patil <[hidden email]> wrote:

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[Vinay Patil]

Hi,

Even tried with ip-address for JobManager.host.name property, but did not work. When I tried netstat -anp | grep 6123 , I see 3 TM connection state as established, however when I submit the job , I see two more entries with state as TIME_WAIT and after some time these entries are gone and I get a Lost to Job Manager Exception.

This only happens when SSL is enabled.

On Thu, Mar 15, 2018 at 10:28 AM, Vinay Patil <[hidden email]> wrote:

Just an update,  I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this.

And in that I am passing the hostname which is same as provided in flink-conf.yaml 

On Thu, Mar 15, 2018 at 7:57 AM, Vinay Patil <[hidden email]> wrote:

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[Chesnay Schepler]

How are you creating the remote environment? In particular, are passing a configuration to the RemoteEnvironment?
Have you set the SSL options in the config?

On 15.03.2018 22:46, Vinay Patil wrote:

Hi,

Even tried with ip-address for JobManager.host.name property, but did not work. When I tried netstat -anp | grep 6123 , I see 3 TM connection state as established, however when I submit the job , I see two more entries with state as TIME_WAIT and after some time these entries are gone and I get a Lost to Job Manager Exception.

This only happens when SSL is enabled.

On Thu, Mar 15, 2018 at 10:28 AM, Vinay Patil <[hidden email]> wrote:

Just an update,  I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this.

And in that I am passing the hostname which is same as provided in flink-conf.yaml 

On Thu, Mar 15, 2018 at 7:57 AM, Vinay Patil <[hidden email]> wrote:

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[Vinay Patil]

Hi,

When I set ssl.verify.hostname to true , the job fails with SSL handshake exception where it tries to match the IP address  instead of the hostname in the certificates. Everything works when I set this to false. The keystore is created with FQDN.

The solution of adding all the hostnames and IP addresses in SAN list is discarded by the company.

And a security concern is raised when I set this parameter to false. I see this https://issues.apache.org/jira/browse/FLINK-5030 in Unresolved state. 

How do Flink support hostname verification ? 

@Chesnay : It would be helpful to know the answer to my previous mail

On Fri, Mar 16, 2018 at 10:15 AM, Vinay Patil <[hidden email]> wrote:

Hi Chesnay,

After setting the configurations for Remote Execution Environment the job gets submitted ,I had to set ssl-verify-hostname to false. 

However, I don't understand why there is a need to do it. I am running the job from master node itself and providing all the configurations in flink-conf.yaml while creating the cluster. So why do I have to copy the same stuff in code ?

On Fri, Mar 16, 2018 at 8:23 AM, Vinay Patil <[hidden email]> wrote:

Hi,

No I am not passing any config to the remote execution environment. I am running the job from master node itself. I have provided SSL configs in flink-xonf.yaml

Do I need to specify any SSL.config as part of Remote Execution env ?

If yes can you please provide me an example.

On Mar 16, 2018 1:56 AM, "Chesnay Schepler [via Apache Flink User Mailing List archive.]" <[hidden email]> wrote:

How are you creating the remote environment? In particular, are passing a configuration to the RemoteEnvironment?
Have you set the SSL options in the config?

On 15.03.2018 22:46, Vinay Patil wrote:

Hi,

Even tried with ip-address for JobManager.host.name property, but did not work. When I tried netstat -anp | grep 6123 , I see 3 TM connection state as established, however when I submit the job , I see two more entries with state as TIME_WAIT and after some time these entries are gone and I get a Lost to Job Manager Exception.

This only happens when SSL is enabled.

On Thu, Mar 15, 2018 at 10:28 AM, Vinay Patil <[hidden email]> wrote:

Just an update,  I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this.

And in that I am passing the hostname which is same as provided in flink-conf.yaml 

On Thu, Mar 15, 2018 at 7:57 AM, Vinay Patil <[hidden email]> wrote:

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]

To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18950.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[Vinay Patil]

Hi,

The passwords are shown in plain text in logs , is this fixed in newer versions of flink (I am using 1.3.2)

Also, please let me know the answer to my previous queries in this mail chain

On Mon, Mar 19, 2018 at 7:35 PM, Vinay Patil <[hidden email]> wrote:

Hi,

When I set ssl.verify.hostname to true , the job fails with SSL handshake exception where it tries to match the IP address  instead of the hostname in the certificates. Everything works when I set this to false. The keystore is created with FQDN.

The solution of adding all the hostnames and IP addresses in SAN list is discarded by the company.

And a security concern is raised when I set this parameter to false. I see this https://issues.apache.org/jira/browse/FLINK-5030 in Unresolved state. 

How do Flink support hostname verification ? 

@Chesnay : It would be helpful to know the answer to my previous mail

On Fri, Mar 16, 2018 at 10:15 AM, Vinay Patil <[hidden email]> wrote:

Hi Chesnay,

After setting the configurations for Remote Execution Environment the job gets submitted ,I had to set ssl-verify-hostname to false. 

However, I don't understand why there is a need to do it. I am running the job from master node itself and providing all the configurations in flink-conf.yaml while creating the cluster. So why do I have to copy the same stuff in code ?

On Fri, Mar 16, 2018 at 8:23 AM, Vinay Patil <[hidden email]> wrote:

Hi,

No I am not passing any config to the remote execution environment. I am running the job from master node itself. I have provided SSL configs in flink-xonf.yaml

Do I need to specify any SSL.config as part of Remote Execution env ?

If yes can you please provide me an example.

On Mar 16, 2018 1:56 AM, "Chesnay Schepler [via Apache Flink User Mailing List archive.]" <[hidden email]> wrote:

How are you creating the remote environment? In particular, are passing a configuration to the RemoteEnvironment?
Have you set the SSL options in the config?

On 15.03.2018 22:46, Vinay Patil wrote:

Hi,

Even tried with ip-address for JobManager.host.name property, but did not work. When I tried netstat -anp | grep 6123 , I see 3 TM connection state as established, however when I submit the job , I see two more entries with state as TIME_WAIT and after some time these entries are gone and I get a Lost to Job Manager Exception.

This only happens when SSL is enabled.

On Thu, Mar 15, 2018 at 10:28 AM, Vinay Patil <[hidden email]> wrote:

Just an update,  I am submitting the job from the master node, not using the normal flink run command to submit the job , but using Remote Execution Environment in code to do this.

And in that I am passing the hostname which is same as provided in flink-conf.yaml 

On Thu, Mar 15, 2018 at 7:57 AM, Vinay Patil <[hidden email]> wrote:

Hi Guys,

Any suggestions here

On Wed, Mar 14, 2018 at 8:08 PM, Vinay Patil <[hidden email]> wrote:

Hi,

After waiting for some time I got the exception as Lost Connection to Job Manager. Message: Could not retrieve the JobExecutionResult from Job Manager

I am submitting the job as remote execution environment. I have specified the exact hostname of JobManager and port as 6123.

Please let me know if any other configurations are needed.

On Wed, Mar 14, 2018 at 11:48 AM, Vinay Patil <[hidden email]> wrote:

Hi Timo,

Not getting any exception , it just says waiting for job completion with a Job ID printed.

On Wed, Mar 14, 2018 at 11:34 AM, Timo Walther [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Hi Vinay,

do you have any exception or log entry that describes the failure?

Regards,
Timo


Am 14.03.18 um 15:51 schrieb Vinay Patil:

Hi,

I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster.

I am not allowed to import the certificate to the java default trustore, so I have provided the trustore and keystore as jvm args to the job.

Is there any other configuration I should do so that the job is submitted

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18909.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]

To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SSL-Setup-on-a-standalone-cluster-tp18907p18950.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML