How to shut down Flink Web Dashboard in detached Yarn session?

[Sai Inampudi]

Hi everyone,

I recently attempted to create a Flink cluster on YARN by executing the following:
~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn

The resulting command was not completely successful but it did end up creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 Job Manager.

When I look at my Yarn Resource Manager, I don't see my application running. CLI calls for the application id also returned nothing.

I would like to kill the existing web dashboard as well as the other lingering task manager/job manager so that I can try recreating the yarn session successfully.

Has anyone encountered this before and has any suggestion? I looked through documentation [1] which says to stop a yarn session, you will want to use the YARN utilities (yarn application -kill <appId>) to stop the YARN session. However, the application id in my logs is not found in the Resource Manager so it seems to already have been killed (due to the original yarn session command not properly executing?).




[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session

[Gary Yao-3]

Hi,

You can use the YARN client to list all applications on your YARN cluster:

    yarn application -list

If this does not show any running applications, the Flink cluster must have
somehow terminated. If you have YARN's log aggregation enabled, you should be
able to view the Flink logs by running:

    yarn logs -applicationId <YOUR_APP_ID>

Best,
Gary

On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]> wrote:

Hi everyone,

I recently attempted to create a Flink cluster on YARN by executing the following:
~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn

The resulting command was not completely successful but it did end up creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 Job Manager.

When I look at my Yarn Resource Manager, I don't see my application running. CLI calls for the application id also returned nothing.

I would like to kill the existing web dashboard as well as the other lingering task manager/job manager so that I can try recreating the yarn session successfully.

Has anyone encountered this before and has any suggestion? I looked through documentation [1] which says to stop a yarn session, you will want to use the YARN utilities (yarn application -kill <appId>) to stop the YARN session. However, the application id in my logs is not found in the Resource Manager so it seems to already have been killed (due to the original yarn session command not properly executing?).




[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session

[Sai Inampudi]

On 2018/12/31 10:53:58, Gary Yao <[hidden email]> wrote:

> Hi,
>
> You can use the YARN client to list all applications on your YARN cluster:
>
>     yarn application -list
>
> If this does not show any running applications, the Flink cluster must have
> somehow terminated. If you have YARN's log aggregation enabled, you should
> be
> able to view the Flink logs by running:
>
>     yarn logs -applicationId <YOUR_APP_ID>
>
> Best,
> Gary
>
> On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > I recently attempted to create a Flink cluster on YARN by executing the
> > following:
> > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn
> >
> > The resulting command was not completely successful but it did end up
> > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1
> > Job Manager.
> >
> > When I look at my Yarn Resource Manager, I don't see my application
> > running. CLI calls for the application id also returned nothing.
> >
> > I would like to kill the existing web dashboard as well as the other
> > lingering task manager/job manager so that I can try recreating the yarn
> > session successfully.
> >
> > Has anyone encountered this before and has any suggestion? I looked
> > through documentation [1] which says to stop a yarn session, you will want
> > to use the YARN utilities (yarn application -kill <appId>) to stop the YARN
> > session. However, the application id in my logs is not found in the
> > Resource Manager so it seems to already have been killed (due to the
> > original yarn session command not properly executing?).
> >
> >
> >
> >
> > [1]
> > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
> >
> Hey Gary, thanks for reaching out.

Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId <YOUR_APP_ID>", I get back nothing (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
/tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)

[Sai Inampudi]

Hey Gary, thanks for reaching out.

Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId <YOUR_APP_ID>", I get back nothing:
    (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
    /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)


(My previous reply was formatted incorrectly so I am replying back with proper formatting. Apologies for the mistake)


On 2018/12/31 18:13:05, Sai Inampudi <[hidden email]> wrote:

>
>
> On 2018/12/31 10:53:58, Gary Yao <[hidden email]> wrote:
> > Hi,
> >
> > You can use the YARN client to list all applications on your YARN cluster:
> >
> >     yarn application -list
> >
> > If this does not show any running applications, the Flink cluster must have
> > somehow terminated. If you have YARN's log aggregation enabled, you should
> > be
> > able to view the Flink logs by running:
> >
> >     yarn logs -applicationId <YOUR_APP_ID>
> >
> > Best,
> > Gary
> >
> > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]> wrote:
> >
> > > Hi everyone,
> > >
> > > I recently attempted to create a Flink cluster on YARN by executing the
> > > following:
> > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn
> > >
> > > The resulting command was not completely successful but it did end up
> > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1
> > > Job Manager.
> > >
> > > When I look at my Yarn Resource Manager, I don't see my application
> > > running. CLI calls for the application id also returned nothing.
> > >
> > > I would like to kill the existing web dashboard as well as the other
> > > lingering task manager/job manager so that I can try recreating the yarn
> > > session successfully.
> > >
> > > Has anyone encountered this before and has any suggestion? I looked
> > > through documentation [1] which says to stop a yarn session, you will want
> > > to use the YARN utilities (yarn application -kill <appId>) to stop the YARN
> > > session. However, the application id in my logs is not found in the
> > > Resource Manager so it seems to already have been killed (due to the
> > > original yarn session command not properly executing?).
> > >
> > >
> > >
> > >
> > > [1]
> > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
> > >
> > Hey Gary, thanks for reaching out.
> Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId <YOUR_APP_ID>", I get back nothing (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
> /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)
>

[Till Rohrmann]

Hi Sai,

could you check that the dashboard you are seeing is really running on Yarn and not a standalone Flink cluster which you have running locally?

Cheers,

Till

On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi <[hidden email]> wrote:

Hey Gary, thanks for reaching out.

Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId <YOUR_APP_ID>", I get back nothing:
    (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
    /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)


(My previous reply was formatted incorrectly so I am replying back with proper formatting. Apologies for the mistake)


On 2018/12/31 18:13:05, Sai Inampudi <[hidden email]> wrote:
>
>
> On 2018/12/31 10:53:58, Gary Yao <[hidden email]> wrote:
> > Hi,
> >
> > You can use the YARN client to list all applications on your YARN cluster:
> >
> >     yarn application -list
> >
> > If this does not show any running applications, the Flink cluster must have
> > somehow terminated. If you have YARN's log aggregation enabled, you should
> > be
> > able to view the Flink logs by running:
> >
> >     yarn logs -applicationId <YOUR_APP_ID>
> >
> > Best,
> > Gary
> >
> > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]> wrote:
> >
> > > Hi everyone,
> > >
> > > I recently attempted to create a Flink cluster on YARN by executing the
> > > following:
> > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn
> > >
> > > The resulting command was not completely successful but it did end up
> > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1
> > > Job Manager.
> > >
> > > When I look at my Yarn Resource Manager, I don't see my application
> > > running. CLI calls for the application id also returned nothing.
> > >
> > > I would like to kill the existing web dashboard as well as the other
> > > lingering task manager/job manager so that I can try recreating the yarn
> > > session successfully.
> > >
> > > Has anyone encountered this before and has any suggestion? I looked
> > > through documentation [1] which says to stop a yarn session, you will want
> > > to use the YARN utilities (yarn application -kill <appId>) to stop the YARN
> > > session. However, the application id in my logs is not found in the
> > > Resource Manager so it seems to already have been killed (due to the
> > > original yarn session command not properly executing?).
> > >
> > >
> > >
> > >
> > > [1]
> > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
> > >
> > Hey Gary, thanks for reaching out.
> Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId <YOUR_APP_ID>", I get back nothing (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
> /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)
>

[Sai Inampudi]

Hey Till,

If it is running on a standalone Flink cluster, wouldn't running stop-cluster.sh work?

When I run stop-cluster.sh, I get back:
No taskexecutor daemon to stop on host <my_host_ip_address>.
No standalonesession daemon to stop on host <my_host_ip_address>.

So I assumed that meant that it is not running on a standalone cluster

On 2019/01/02 14:13:52, Till Rohrmann <[hidden email]> wrote:

> Hi Sai,
>
> could you check that the dashboard you are seeing is really running on Yarn
> and not a standalone Flink cluster which you have running locally?
>
> Cheers,
> Till
>
> On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi <[hidden email]> wrote:
>
> > Hey Gary, thanks for reaching out.
> >
> > Executing "yarn application -list" does not return my flink cluster so I
> > assume like in my initial post that the application must be terminated. My
> > config when I ran the job did not have log aggregation enabled and that
> > might be why when I try to look at the logs via "yarn logs -applicationId
> > <YOUR_APP_ID>", I get back nothing:
> >     (e.g. Unable to get ApplicationState. Attempting to fetch logs
> > directly from the filesystem.
> >     /tmp/logs/si022833/logs/application_1545041832015_73428 does not
> > exist.)
> >
> >
> > (My previous reply was formatted incorrectly so I am replying back with
> > proper formatting. Apologies for the mistake)
> >
> >
> > On 2018/12/31 18:13:05, Sai Inampudi <[hidden email]> wrote:
> > >
> > >
> > > On 2018/12/31 10:53:58, Gary Yao <[hidden email]> wrote:
> > > > Hi,
> > > >
> > > > You can use the YARN client to list all applications on your YARN
> > cluster:
> > > >
> > > >     yarn application -list
> > > >
> > > > If this does not show any running applications, the Flink cluster must
> > have
> > > > somehow terminated. If you have YARN's log aggregation enabled, you
> > should
> > > > be
> > > > able to view the Flink logs by running:
> > > >
> > > >     yarn logs -applicationId <YOUR_APP_ID>
> > > >
> > > > Best,
> > > > Gary
> > > >
> > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]>
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I recently attempted to create a Flink cluster on YARN by executing
> > the
> > > > > following:
> > > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm
> > flink_yarn
> > > > >
> > > > > The resulting command was not completely successful but it did end up
> > > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot,
> > and 1
> > > > > Job Manager.
> > > > >
> > > > > When I look at my Yarn Resource Manager, I don't see my application
> > > > > running. CLI calls for the application id also returned nothing.
> > > > >
> > > > > I would like to kill the existing web dashboard as well as the other
> > > > > lingering task manager/job manager so that I can try recreating the
> > yarn
> > > > > session successfully.
> > > > >
> > > > > Has anyone encountered this before and has any suggestion? I looked
> > > > > through documentation [1] which says to stop a yarn session, you
> > will want
> > > > > to use the YARN utilities (yarn application -kill <appId>) to stop
> > the YARN
> > > > > session. However, the application id in my logs is not found in the
> > > > > Resource Manager so it seems to already have been killed (due to the
> > > > > original yarn session command not properly executing?).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
> > > > >
> > > > Hey Gary, thanks for reaching out.
> > > Executing "yarn application -list" does not return my flink cluster so I
> > assume like in my initial post that the application must be terminated. My
> > config when I ran the job did not have log aggregation enabled and that
> > might be why when I try to look at the logs via "yarn logs -applicationId
> > <YOUR_APP_ID>", I get back nothing (e.g. Unable to get ApplicationState.
> > Attempting to fetch logs directly from the filesystem.
> > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)
> > >
> >
>

[Till Rohrmann]

You could also use `jsp` or `ps` to check that no TaskExecutor and StandaloneJobClusterEntrypoint is running. If there are no such processes, then there should not be a Flink cluster running locally.

Cheers,

Till

On Wed, Jan 2, 2019 at 6:31 PM Sai Inampudi <[hidden email]> wrote:

Hey Till,

If it is running on a standalone Flink cluster, wouldn't running stop-cluster.sh work?

When I run stop-cluster.sh, I get back:
No taskexecutor daemon to stop on host <my_host_ip_address>.
No standalonesession daemon to stop on host <my_host_ip_address>.

So I assumed that meant that it is not running on a standalone cluster

On 2019/01/02 14:13:52, Till Rohrmann <[hidden email]> wrote:
> Hi Sai,
>
> could you check that the dashboard you are seeing is really running on Yarn
> and not a standalone Flink cluster which you have running locally?
>
> Cheers,
> Till
>
> On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi <[hidden email]> wrote:
>
> > Hey Gary, thanks for reaching out.
> >
> > Executing "yarn application -list" does not return my flink cluster so I
> > assume like in my initial post that the application must be terminated. My
> > config when I ran the job did not have log aggregation enabled and that
> > might be why when I try to look at the logs via "yarn logs -applicationId
> > <YOUR_APP_ID>", I get back nothing:
> >     (e.g. Unable to get ApplicationState. Attempting to fetch logs
> > directly from the filesystem.
> >     /tmp/logs/si022833/logs/application_1545041832015_73428 does not
> > exist.)
> >
> >
> > (My previous reply was formatted incorrectly so I am replying back with
> > proper formatting. Apologies for the mistake)
> >
> >
> > On 2018/12/31 18:13:05, Sai Inampudi <[hidden email]> wrote:
> > >
> > >
> > > On 2018/12/31 10:53:58, Gary Yao <[hidden email]> wrote:
> > > > Hi,
> > > >
> > > > You can use the YARN client to list all applications on your YARN
> > cluster:
> > > >
> > > >     yarn application -list
> > > >
> > > > If this does not show any running applications, the Flink cluster must
> > have
> > > > somehow terminated. If you have YARN's log aggregation enabled, you
> > should
> > > > be
> > > > able to view the Flink logs by running:
> > > >
> > > >     yarn logs -applicationId <YOUR_APP_ID>
> > > >
> > > > Best,
> > > > Gary
> > > >
> > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi <[hidden email]>
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I recently attempted to create a Flink cluster on YARN by executing
> > the
> > > > > following:
> > > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm
> > flink_yarn
> > > > >
> > > > > The resulting command was not completely successful but it did end up
> > > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot,
> > and 1
> > > > > Job Manager.
> > > > >
> > > > > When I look at my Yarn Resource Manager, I don't see my application
> > > > > running. CLI calls for the application id also returned nothing.
> > > > >
> > > > > I would like to kill the existing web dashboard as well as the other
> > > > > lingering task manager/job manager so that I can try recreating the
> > yarn
> > > > > session successfully.
> > > > >
> > > > > Has anyone encountered this before and has any suggestion? I looked
> > > > > through documentation [1] which says to stop a yarn session, you
> > will want
> > > > > to use the YARN utilities (yarn application -kill <appId>) to stop
> > the YARN
> > > > > session. However, the application id in my logs is not found in the
> > > > > Resource Manager so it seems to already have been killed (due to the
> > > > > original yarn session command not properly executing?).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
> > > > >
> > > > Hey Gary, thanks for reaching out.
> > > Executing "yarn application -list" does not return my flink cluster so I
> > assume like in my initial post that the application must be terminated. My
> > config when I ran the job did not have log aggregation enabled and that
> > might be why when I try to look at the logs via "yarn logs -applicationId
> > <YOUR_APP_ID>", I get back nothing (e.g. Unable to get ApplicationState.
> > Attempting to fetch logs directly from the filesystem.
> > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)
> > >
> >
>