(DEPRECATED) Apache Flink User Mailing List archive.

Documentation for translation of Job graph to Execution graph

Classic

List

Threaded

7 messages Options

Bajaj, Abhinav

Documentation for translation of Job graph to Execution graph

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

Aljoscha Krettek

Re: Documentation for translation of Job graph to Execution graph

Hi,

I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,

Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <[hidden email]> wrote:

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

Bajaj, Abhinav

Re: Documentation for translation of Job graph to Execution graph

Hi,

Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it.

In general, there are two things I am trying to understand and get comfortable with -

How a Job graph is translated to Execution graph. The logs and monitoring APIs are for the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap.
The job manager & task manager logs are tricky to decipher. Especially when there are multiple jobs running. Is there a way to filter the logs for a single job ?

~ Abhi

From: Aljoscha Krettek <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, June 17, 2016 at 2:31 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,

I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,

Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <[hidden email]> wrote:

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

rmetzger0

Re: Documentation for translation of Job graph to Execution graph

Hi,

the link has been added newly, yes.

Regarding Q1, since there is no documentation right now, I have to refer you to our code. In the JobManager.scala class, there is a method "private def submitJob(jobGraph, ...") where the ExecutionGraph is created. I think that's a good starting point for looking through the code. (I also added Till to the message if he wants to chime in)

Q2: Currently, Flink doesn't add the job name to the logs, so its indeed not very easy to separate the log entries generated by different jobs. In general, we recommend running one JobManager per job (multiple jobs is of course also supported).

On Sat, Jun 18, 2016 at 1:41 AM, Bajaj, Abhinav <[hidden email]> wrote:

Hi,

Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it.

In general, there are two things I am trying to understand and get comfortable with -

How a Job graph is translated to Execution graph. The logs and monitoring APIs are for the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap.
The job manager & task manager logs are tricky to decipher. Especially when there are multiple jobs running. Is there a way to filter the logs for a single job ?

~ Abhi

From: Aljoscha Krettek <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, June 17, 2016 at 2:31 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,
I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,

Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <[hidden email]> wrote:

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

Bajaj, Abhinav

Re: Documentation for translation of Job graph to Execution graph

Thanks Robert for helpful reply.

I have follow up on the Q2 - "In general, we recommend running one JobManager per job”

I understand how this can be achieved while running in Yarn, I.e. by submitting single Flink Jobs.

Is their some other way of setting Flink to configure single Jobmanager per job ?

From: Robert Metzger <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, June 21, 2016 at 8:23 AM
To: "[hidden email]" <[hidden email]>, Till Rohrmann <[hidden email]>
Cc: Aljoscha Krettek <[hidden email]>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,

the link has been added newly, yes.

On Sat, Jun 18, 2016 at 1:41 AM, Bajaj, Abhinav <[hidden email]> wrote:

Hi,

Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it.

In general, there are two things I am trying to understand and get comfortable with -

How a Job graph is translated to Execution graph. The logs and monitoring APIs are for the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap.
The job manager & task manager logs are tricky to decipher. Especially when there are multiple jobs running. Is there a way to filter the logs for a single job ?

~ Abhi

From: Aljoscha Krettek <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, June 17, 2016 at 2:31 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,
I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,

Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <[hidden email]> wrote:

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

Bajaj, Abhinav

Re: Documentation for translation of Job graph to Execution graph

In reply to this post by rmetzger0

Hi Robert,

Thanks for helpful reply.

I have couple of follow up questions on your reply - "In general, we recommend running one JobManager per job”

I understand how this can be achieved while running in Yarn, I.e. by submitting single Flink Jobs.

Is their some other way of setting Flink to configure single Jobmanager per job ?

Is their a plan to add the Job id or name to the logs ?

Thanks,

Abhi

Hi,

the link has been added newly, yes.

On Sat, Jun 18, 2016 at 1:41 AM, Bajaj, Abhinav <[hidden email]> wrote:

Hi,

Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it.

In general, there are two things I am trying to understand and get comfortable with -

How a Job graph is translated to Execution graph. The logs and monitoring APIs are for the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap.
The job manager & task manager logs are tricky to decipher. Especially when there are multiple jobs running. Is there a way to filter the logs for a single job ?

~ Abhi

From: Aljoscha Krettek <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, June 17, 2016 at 2:31 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,
I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,

Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <[hidden email]> wrote:

Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.

I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,

Abhi

Ufuk Celebi

Re: Documentation for translation of Job graph to Execution graph

On Wed, Jun 29, 2016 at 9:19 PM, Bajaj, Abhinav <[hidden email]> wrote:
> Is their a plan to add the Job id or name to the logs ?

This is now part of the YARN client output and should be part of the
1.1 release.

Regarding your other question: in standalone mode, you have to
manually make sure to not submit multiple jobs to a single cluster.