how to submit jobs remotely when a rest proxy like nginx is used and REST endpoint is bind to loopback interface?

[Ming Li]

Hi,

The flink official document clearly states that "Simple mutual authentication may be enabled by configuration if authentication of connections to the REST endpoint is required, but we recommend to deploy a “side car proxy”: Bind the REST endpoint to the loopback interface (or the pod-local interface in Kubernetes) and start a REST proxy that authenticates and forwards the requests to Flink. Examples for proxies that Flink users have deployed are Envoy Proxy or NGINX with MOD_AUTH."

 

So I am wondering, in standalone mode when HA is not enabeld, when a rest proxy like nginx is used, and rest endpoint is bind to the loopback interface, how should we submit jobs remotely? 

 

ps.

1. sample flink-conf.yaml settings, and nginx settings are as below showing:

rest.bind-port: 9091/rest.bind-address: 127.0.0.1 （this is the port and ip where the rest endpoint bind itself to in the host where it is started）  

rest.port: 9091/rest.address: 127.0.0.1 (this is the port and ip used by rest clients when submit requests, so basically it should reach the above  rest.bind-port/rest.bind-address) 

2. I know that we can use curl to request the nginx proxy, which authenticates and forwards the request to flink, as below showing: curl -v -u user1:user1 http://10.20.39.43:8080/config (which is the address where nginx is listening to)

3. I know that  we can submit jobs from the host where job manager is located, as below showing: 

/opt/flink-1.12.2/bin/flink run -m 127.0.0.1:9091 /opt/flink-1.12.2/examples/batch/WordCount.jar --input /tmp/README.txt --output /tmp/flink.test.txt11  ()

Thanks!

--

[Arvid Heise-4]

Hi Ming,

instead of using the command line interface to run Flink applications, you should use the REST API [1].

You should first upload your jar in one call and then execute the job in the second call.

The rest endpoint would be http://10.20.39.43:8080/

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/rest_api.html#jars-jarid-run

On Fri, Apr 9, 2021 at 4:45 AM Ming Li <[hidden email]> wrote:

Hi,

The flink official document clearly states that "Simple mutual authentication may be enabled by configuration if authentication of connections to the REST endpoint is required, but we recommend to deploy a “side car proxy”: Bind the REST endpoint to the loopback interface (or the pod-local interface in Kubernetes) and start a REST proxy that authenticates and forwards the requests to Flink. Examples for proxies that Flink users have deployed are Envoy Proxy or NGINX with MOD_AUTH."

 

So I am wondering, in standalone mode when HA is not enabeld, when a rest proxy like nginx is used, and rest endpoint is bind to the loopback interface, how should we submit jobs remotely? 

 

ps.

1. sample flink-conf.yaml settings, and nginx settings are as below showing:

rest.bind-port: 9091/rest.bind-address: 127.0.0.1 （this is the port and ip where the rest endpoint bind itself to in the host where it is started）  

rest.port: 9091/rest.address: 127.0.0.1 (this is the port and ip used by rest clients when submit requests, so basically it should reach the above  rest.bind-port/rest.bind-address) 

2. I know that we can use curl to request the nginx proxy, which authenticates and forwards the request to flink, as below showing: curl -v -u user1:user1 http://10.20.39.43:8080/config (which is the address where nginx is listening to)

3. I know that  we can submit jobs from the host where job manager is located, as below showing: 

/opt/flink-1.12.2/bin/flink run -m 127.0.0.1:9091 /opt/flink-1.12.2/examples/batch/WordCount.jar --input /tmp/README.txt --output /tmp/flink.test.txt11  ()

Thanks!

--

Best Regards
Michael Li