Jars replication

[Andrés Garagiola]

Hi,

I'm configuring a Flink cluster with high availability based on ZooKeeper and two Job Managers.

When I upload a jar using the /jars/upload REST API, I don't get the jar replicated in both JMs. Is this the expected behavior?

I want to configure the cluster in such a way that once the jar is uploaded, I'm able to schedule multiple jobs based on the same jar id, independently of what is the leader JM at the scheduling time.

Also, I'm using a K8 service to reach the job managers, this service choose one of the JM pods in every request, should I invoke always the leader one?

Thanks

[Chesnay Schepler]

Jars are not replicated to all JobManagers, this is currently expected,
but generally undesirable for the use-case you out-lined.

IIRC the important part is that the upload goes directly against the
leader, the run request can be sent anywhere and it will be redirected
internally to the leader.

On 17/04/2020 09:59, Andrés Garagiola wrote:

> Hi,
>
> I'm configuring a Flink cluster with high availability based on
> ZooKeeper and two Job Managers.
>
> When I upload a jar using the /jars/upload REST API, I don't get the
> jar replicated in both JMs. Is this the expected behavior?
>
> I want to configure the cluster in such a way that once the jar is
> uploaded, I'm able to schedule multiple jobs based on the same jar id,
> independently of what is the leader JM at the scheduling time.
>
> Also, I'm using a K8 service to reach the job managers, this service
> choose one of the JM pods in every request, should I invoke always the
> leader one?
>
> Thanks

[Chesnay Schepler]

My apologies, I remembered wrong. The jar endpoints all require working against the leading job master unfortunately.

On 17/04/2020 10:18, Andrés Garagiola wrote:

Thank Chesnay,

I'm invoking the "/jars" endpoint in both JMs and only one of them answers with the uploaded jar. If I try to send a run request to the one that doesn't have the jar, it responds with a "jar not found" error. I didn't find documentation about this, so I don't know if it is the expected behavior or something wrong in my cluster configuration.

Regards

On Fri, Apr 17, 2020 at 10:05 AM Chesnay Schepler <[hidden email]> wrote:

Jars are not replicated to all JobManagers, this is currently expected,
but generally undesirable for the use-case you out-lined.

IIRC the important part is that the upload goes directly against the
leader, the run request can be sent anywhere and it will be redirected
internally to the leader.

On 17/04/2020 09:59, Andrés Garagiola wrote:
> Hi,
>
> I'm configuring a Flink cluster with high availability based on
> ZooKeeper and two Job Managers.
>
> When I upload a jar using the /jars/upload REST API, I don't get the
> jar replicated in both JMs. Is this the expected behavior?
>
> I want to configure the cluster in such a way that once the jar is
> uploaded, I'm able to schedule multiple jobs based on the same jar id,
> independently of what is the leader JM at the scheduling time.
>
> Also, I'm using a K8 service to reach the job managers, this service
> choose one of the JM pods in every request, should I invoke always the
> leader one?
>
> Thanks