Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[sidhant gupta]

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper.

As per [1] , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs?

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks 

Sidhant Gupta

[Yang Wang]

I think the Flink client could make a connection with ZooKeeper via the network load balancer.

Flink client is not aware of whether it is a network balancer or multiple ZooKeeper server address.

After then Flink client will retrieve the active leader JobManager address via ZooKeeperHAService 

and submit the job successfully via rest client.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月2日周二 下午11:14写道：

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper.

As per [1] , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs?

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks 

Sidhant Gupta

[sidhant gupta]

Is it possible to use flink CLI instead of flink client for connecting zookeeper using network load balancer to retrieve the leader Jobmanager address?

On Wed, Feb 3, 2021, 12:42 PM Yang Wang <[hidden email]> wrote:

I think the Flink client could make a connection with ZooKeeper via the network load balancer.

Flink client is not aware of whether it is a network balancer or multiple ZooKeeper server address.

After then Flink client will retrieve the active leader JobManager address via ZooKeeperHAService 

and submit the job successfully via rest client.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月2日周二 下午11:14写道：

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper.

As per [1] , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs?

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks 

Sidhant Gupta

[Yang Wang]

Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session ...) for the job management,

it will eventually call the RestClusterClient, which could retrieve the leader JobManager address from ZK.

Please ensure that you have specified the HA related config options in CLI via -D or set them in the flink-conf.yaml.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月3日周三 下午10:02写道：

Is it possible to use flink CLI instead of flink client for connecting zookeeper using network load balancer to retrieve the leader Jobmanager address?

On Wed, Feb 3, 2021, 12:42 PM Yang Wang <[hidden email]> wrote:

I think the Flink client could make a connection with ZooKeeper via the network load balancer.

Flink client is not aware of whether it is a network balancer or multiple ZooKeeper server address.

After then Flink client will retrieve the active leader JobManager address via ZooKeeperHAService 

and submit the job successfully via rest client.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月2日周二 下午11:14写道：

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper.

As per [1] , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs?

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks 

Sidhant Gupta

[sidhant gupta]

Thanks Yang for your help. 

On Thu, Feb 4, 2021, 8:28 AM Yang Wang <[hidden email]> wrote:

Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session ...) for the job management,

it will eventually call the RestClusterClient, which could retrieve the leader JobManager address from ZK.

Please ensure that you have specified the HA related config options in CLI via -D or set them in the flink-conf.yaml.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月3日周三 下午10:02写道：

Is it possible to use flink CLI instead of flink client for connecting zookeeper using network load balancer to retrieve the leader Jobmanager address?

On Wed, Feb 3, 2021, 12:42 PM Yang Wang <[hidden email]> wrote:

I think the Flink client could make a connection with ZooKeeper via the network load balancer.

Flink client is not aware of whether it is a network balancer or multiple ZooKeeper server address.

After then Flink client will retrieve the active leader JobManager address via ZooKeeperHAService 

and submit the job successfully via rest client.

Best,

Yang

sidhant gupta <[hidden email]> 于2021年2月2日周二 下午11:14写道：

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper.

As per [1] , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs?

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks 

Sidhant Gupta