Specify jobmanager port in HA mode

[Brian Chhun]

Hello,

Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?

Thanks,

Brian

[Ufuk Celebi]

Hey Brian,

I think that it is currently not possible. I will look into whether there is a workaround. In any case, this sounds like a useful thing and it shouldn’t be too complicated to add the desired behaviour.

I’ve opened an issue [1] for it and will look into it tomorrow.

Is this currently blocking you from using Flink?

– Ufuk

[1] https://issues.apache.org/jira/browse/FLINK-3172

> On 14 Dec 2015, at 20:33, Brian Chhun <[hidden email]> wrote:
>
> Hello,
>
> Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?
>
> Thanks,
> Brian

[Ufuk Celebi]

This has been recently added to the YARN client by Robert [1]: https://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html#running-flink-on-yarn-behind-firewalls

Are you running YARN?

– Ufuk

[1] https://github.com/apache/flink/pull/1416

> On 15 Dec 2015, at 00:03, Ufuk Celebi <[hidden email]> wrote:
>
> Hey Brian,
>
> I think that it is currently not possible. I will look into whether there is a workaround. In any case, this sounds like a useful thing and it shouldn’t be too complicated to add the desired behaviour.
>
> I’ve opened an issue [1] for it and will look into it tomorrow.
>
> Is this currently blocking you from using Flink?
>
> – Ufuk
>
> [1] https://issues.apache.org/jira/browse/FLINK-3172
>
>> On 14 Dec 2015, at 20:33, Brian Chhun <[hidden email]> wrote:
>>
>> Hello,
>>
>> Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?
>>
>> Thanks,
>> Brian
>

[Cory Monty]

Ufuk,

I'm a colleague of Brian. Unfortunately, we are not running YARN so I don't think that PR applies to us. We're trying to run a standalone cluster.

Cheers,

Cory

On Mon, Dec 14, 2015 at 5:23 PM, Ufuk Celebi <[hidden email]> wrote:

This has been recently added to the YARN client by Robert [1]: https://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html#running-flink-on-yarn-behind-firewalls

Are you running YARN?

– Ufuk

[1] https://github.com/apache/flink/pull/1416

> On 15 Dec 2015, at 00:03, Ufuk Celebi <[hidden email]> wrote:
>
> Hey Brian,
>
> I think that it is currently not possible. I will look into whether there is a workaround. In any case, this sounds like a useful thing and it shouldn’t be too complicated to add the desired behaviour.
>
> I’ve opened an issue [1] for it and will look into it tomorrow.
>
> Is this currently blocking you from using Flink?
>
> – Ufuk
>
> [1] https://issues.apache.org/jira/browse/FLINK-3172
>
>> On 14 Dec 2015, at 20:33, Brian Chhun <[hidden email]> wrote:
>>
>> Hello,
>>
>> Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?
>>
>> Thanks,
>> Brian
>

[Ufuk Celebi]

Hey Cory,

I’ve opened a pull request for this [1]. It’s based on the current snapshot version.

This is not a bug fix, but since we are only adding behaviour I think that it can make its way into 0.10.2 if it is a blocker for users.

Currently, you can only try it out by building it yourself. Is this an option? If yes, [2] is a guide on how to do it. Just replace the clone command with

git clone -b 3172-ha_jm_port https://github.com/uce/flink.git

– Ufuk

[1] https://github.com/apache/flink/pull/1458
[2] https://ci.apache.org/projects/flink/flink-docs-master/setup/building.html

> On 15 Dec 2015, at 00:28, Cory Monty <[hidden email]> wrote:
>
> Ufuk,
>
> I'm a colleague of Brian. Unfortunately, we are not running YARN so I don't think that PR applies to us. We're trying to run a standalone cluster.
>
> Cheers,
>
> Cory
>
> On Mon, Dec 14, 2015 at 5:23 PM, Ufuk Celebi <[hidden email]> wrote:
> This has been recently added to the YARN client by Robert [1]: https://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html#running-flink-on-yarn-behind-firewalls
>
> Are you running YARN?
>
> – Ufuk
>
> [1] https://github.com/apache/flink/pull/1416
>
> > On 15 Dec 2015, at 00:03, Ufuk Celebi <[hidden email]> wrote:
> >
> > Hey Brian,
> >
> > I think that it is currently not possible. I will look into whether there is a workaround. In any case, this sounds like a useful thing and it shouldn’t be too complicated to add the desired behaviour.
> >
> > I’ve opened an issue [1] for it and will look into it tomorrow.
> >
> > Is this currently blocking you from using Flink?
> >
> > – Ufuk
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-3172
> >
> >> On 14 Dec 2015, at 20:33, Brian Chhun <[hidden email]> wrote:
> >>
> >> Hello,
> >>
> >> Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?
> >>
> >> Thanks,
> >> Brian
> >
>
>

[Cory Monty]

Hey Ufuk,

We can try building it, but it might be a week or so given the holidays. We're currently targeting development against 1.0.0-SNAPSHOT and that release is OK unless 1.0 release is more than 2-3 months away.

Let me know if you still want us to build your PR and test it out.

Cheers,

Cory

On Wed, Dec 16, 2015 at 7:51 AM, Ufuk Celebi <[hidden email]> wrote:

Hey Cory,

I’ve opened a pull request for this [1]. It’s based on the current snapshot version.

This is not a bug fix, but since we are only adding behaviour I think that it can make its way into 0.10.2 if it is a blocker for users.

Currently, you can only try it out by building it yourself. Is this an option? If yes, [2] is a guide on how to do it. Just replace the clone command with

git clone -b 3172-ha_jm_port https://github.com/uce/flink.git

– Ufuk

[1] https://github.com/apache/flink/pull/1458
[2] https://ci.apache.org/projects/flink/flink-docs-master/setup/building.html

> On 15 Dec 2015, at 00:28, Cory Monty <[hidden email]> wrote:
>
> Ufuk,
>
> I'm a colleague of Brian. Unfortunately, we are not running YARN so I don't think that PR applies to us. We're trying to run a standalone cluster.
>
> Cheers,
>
> Cory
>
> On Mon, Dec 14, 2015 at 5:23 PM, Ufuk Celebi <[hidden email]> wrote:
> This has been recently added to the YARN client by Robert [1]: https://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html#running-flink-on-yarn-behind-firewalls
>
> Are you running YARN?
>
> – Ufuk
>
> [1] https://github.com/apache/flink/pull/1416
>
> > On 15 Dec 2015, at 00:03, Ufuk Celebi <[hidden email]> wrote:
> >
> > Hey Brian,
> >
> > I think that it is currently not possible. I will look into whether there is a workaround. In any case, this sounds like a useful thing and it shouldn’t be too complicated to add the desired behaviour.
> >
> > I’ve opened an issue [1] for it and will look into it tomorrow.
> >
> > Is this currently blocking you from using Flink?
> >
> > – Ufuk
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-3172
> >
> >> On 14 Dec 2015, at 20:33, Brian Chhun <[hidden email]> wrote:
> >>
> >> Hello,
> >>
> >> Is it possible to set the job manager rpc port when running in HA mode? Or is there a workaround or solution if we're running task managers with a firewall?
> >>
> >> Thanks,
> >> Brian
> >
>
>

[Ufuk Celebi]

> On 17 Dec 2015, at 19:36, Cory Monty <[hidden email]> wrote:
>
> Hey Ufuk,
>
> We can try building it, but it might be a week or so given the holidays. We're currently targeting development against 1.0.0-SNAPSHOT and that release is OK unless 1.0 release is more than 2-3 months away.

OK, that’s the expected timeframe for the 1.0.0 release.

> Let me know if you still want us to build your PR and test it out.

The change should be in after the Holidays, so it should be OK to wait if you don’t need it now.

– Ufuk

[Ufuk Celebi]

This is now part of the master branch and should be part of the SNAPSHOT builds soon. The HA docs have a short paragraph on how to configure it.

– Ufuk

> On 21 Dec 2015, at 12:10, Ufuk Celebi <[hidden email]> wrote:
>
>
>> On 17 Dec 2015, at 19:36, Cory Monty <[hidden email]> wrote:
>>
>> Hey Ufuk,
>>
>> We can try building it, but it might be a week or so given the holidays. We're currently targeting development against 1.0.0-SNAPSHOT and that release is OK unless 1.0 release is more than 2-3 months away.
>
> OK, that’s the expected timeframe for the 1.0.0 release.
>
>> Let me know if you still want us to build your PR and test it out.
>
> The change should be in after the Holidays, so it should be OK to wait if you don’t need it now.
>
> – Ufuk