CPU Cores of JobManager
Locked
 
|
Hi
Now I am looking for the way to increase the number of allocated CPU cores because my JobManagaer WEBUI is very heavy and sometimes freeze. I think this is caused by the resource shortage of JobManager. How can I increase the number of CPU for JobManager in YARN mode? Thanks Yuta -- |
|
|
Hi Yuta,
as far as I know you cannot assign more cores to a JobManager. Can you tell us a bit more about your environment? How many jobs does the JobManager has to manage? How much heap memory is assigned to the JobManager? Maybe you can use a profiler and find out which component consumes so much CPU resources? Regards, Timo Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: > Hi > > Now I am looking for the way to increase the number of allocated CPU > cores because my JobManagaer WEBUI is very heavy and sometimes freeze. > > I think this is caused by the resource shortage of JobManager. > How can I increase the number of CPU for JobManager in YARN mode? > > > > Thanks > Yuta |
|
|
Hi Timo
Thank you for your early reply. These are commands which I run my apps. ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 ./bin/flink run -p 100 <jar1> <jar1 parameters> ./bin/flink run -p 100 <jar2> <jar2 parameters> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. > Maybe you can use a profiler and find out which component consumes so > much CPU resources? You mean Java Flight Recorder or JITWatch? Or, Flink has original profiler? https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html Regards, Yuta On 2017/12/05 18:02, Timo Walther wrote: > Hi Yuta, > > as far as I know you cannot assign more cores to a JobManager. > > Can you tell us a bit more about your environment? How many jobs does > the JobManager has to manage? How much heap memory is assigned to the > JobManager? > > Maybe you can use a profiler and find out which component consumes so > much CPU resources? > > Regards, > Timo > > > Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >> Hi >> >> Now I am looking for the way to increase the number of allocated CPU >> cores because my JobManagaer WEBUI is very heavy and sometimes freeze. >> >> I think this is caused by the resource shortage of JobManager. >> How can I increase the number of CPU for JobManager in YARN mode? >> >> >> >> Thanks >> Yuta > > > |
|
|
I had some profiling tool like jvisualvm in mind. Are you executing
streaming or batch jobs? If streaming, is checkpointing enabled and which type of statebackend? @Chesnay do you have experience with slow behavior of the Web UI? Regards, Timo Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: > Hi Timo > > Thank you for your early reply. > > These are commands which I run my apps. > ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 > ./bin/flink run -p 100 <jar1> <jar1 parameters> > ./bin/flink run -p 100 <jar2> <jar2 parameters> > > So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. > > > Maybe you can use a profiler and find out which component consumes so > > much CPU resources? > You mean Java Flight Recorder or JITWatch? > Or, Flink has original profiler? > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html > > > Regards, > Yuta > > On 2017/12/05 18:02, Timo Walther wrote: >> Hi Yuta, >> >> as far as I know you cannot assign more cores to a JobManager. >> >> Can you tell us a bit more about your environment? How many jobs does >> the JobManager has to manage? How much heap memory is assigned to the >> JobManager? >> >> Maybe you can use a profiler and find out which component consumes so >> much CPU resources? >> >> Regards, >> Timo >> >> >> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >>> Hi >>> >>> Now I am looking for the way to increase the number of allocated CPU >>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze. >>> >>> I think this is caused by the resource shortage of JobManager. >>> How can I increase the number of CPU for JobManager in YARN mode? >>> >>> >>> >>> Thanks >>> Yuta >> >> >> |
|
|
Hi Timo
I execute streaming job without checkpointing and I don't configure any state backend, so it may be "MemoryStateBackend". Actually, my streaming app just reads data from kafka and writes it to an external DB. Its not so complicated. Regards, Yuta On 2017/12/05 19:55, Timo Walther wrote: > I had some profiling tool like jvisualvm in mind. Are you executing > streaming or batch jobs? If streaming, is checkpointing enabled and > which type of statebackend? > > @Chesnay do you have experience with slow behavior of the Web UI? > > Regards, > Timo > > > Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: >> Hi Timo >> >> Thank you for your early reply. >> >> These are commands which I run my apps. >> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 >> ./bin/flink run -p 100 <jar1> <jar1 parameters> >> ./bin/flink run -p 100 <jar2> <jar2 parameters> >> >> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. >> >> > Maybe you can use a profiler and find out which component consumes so >> > much CPU resources? >> You mean Java Flight Recorder or JITWatch? >> Or, Flink has original profiler? >> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html >> >> >> Regards, >> Yuta >> >> On 2017/12/05 18:02, Timo Walther wrote: >>> Hi Yuta, >>> >>> as far as I know you cannot assign more cores to a JobManager. >>> >>> Can you tell us a bit more about your environment? How many jobs does >>> the JobManager has to manage? How much heap memory is assigned to the >>> JobManager? >>> >>> Maybe you can use a profiler and find out which component consumes so >>> much CPU resources? >>> >>> Regards, >>> Timo >>> >>> >>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >>>> Hi >>>> >>>> Now I am looking for the way to increase the number of allocated CPU >>>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze. >>>> >>>> I think this is caused by the resource shortage of JobManager. >>>> How can I increase the number of CPU for JobManager in YARN mode? >>>> >>>> >>>> >>>> Thanks >>>> Yuta >>> >>> >>> > > |
|
|
Which version of Fllink are you using?
Is the high CPU usage caused by the browser or the JVM? One thing we should investigate is how many requests the UI sends to the JobManager, but I don't know of a good way to do that without modifying the source. On 06.12.2017 01:20, Yuta Morisawa wrote: > Hi Timo > > I execute streaming job without checkpointing and I don't configure > any state backend, so it may be "MemoryStateBackend". > > Actually, my streaming app just reads data from kafka and writes it to > an external DB. Its not so complicated. > > Regards, > Yuta > > On 2017/12/05 19:55, Timo Walther wrote: >> I had some profiling tool like jvisualvm in mind. Are you executing >> streaming or batch jobs? If streaming, is checkpointing enabled and >> which type of statebackend? >> >> @Chesnay do you have experience with slow behavior of the Web UI? >> >> Regards, >> Timo >> >> >> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: >>> Hi Timo >>> >>> Thank you for your early reply. >>> >>> These are commands which I run my apps. >>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 >>> ./bin/flink run -p 100 <jar1> <jar1 parameters> >>> ./bin/flink run -p 100 <jar2> <jar2 parameters> >>> >>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. >>> >>> > Maybe you can use a profiler and find out which component consumes so >>> > much CPU resources? >>> You mean Java Flight Recorder or JITWatch? >>> Or, Flink has original profiler? >>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html >>> >>> >>> Regards, >>> Yuta >>> >>> On 2017/12/05 18:02, Timo Walther wrote: >>>> Hi Yuta, >>>> >>>> as far as I know you cannot assign more cores to a JobManager. >>>> >>>> Can you tell us a bit more about your environment? How many jobs >>>> does the JobManager has to manage? How much heap memory is assigned >>>> to the JobManager? >>>> >>>> Maybe you can use a profiler and find out which component consumes >>>> so much CPU resources? >>>> >>>> Regards, >>>> Timo >>>> >>>> >>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >>>>> Hi >>>>> >>>>> Now I am looking for the way to increase the number of allocated >>>>> CPU cores because my JobManagaer WEBUI is very heavy and sometimes >>>>> freeze. >>>>> >>>>> I think this is caused by the resource shortage of JobManager. >>>>> How can I increase the number of CPU for JobManager in YARN mode? >>>>> >>>>> >>>>> >>>>> Thanks >>>>> Yuta >>>> >>>> >>>> >> >> > > |
|
|
> Which version of Fllink are you using?
I'm using Flink 1.3.0. > Is the high CPU usage caused by the browser or the JVM? The CPU usage of browser is not high, the JVM is not also. I found that if I remove the rebalance function in my code, the WEBUI works well. I don't know why but I think rebalance function imposes a load on JobManager. Regards, Yuta On 2017/12/06 17:08, Chesnay Schepler wrote: > Which version of Fllink are you using? > > Is the high CPU usage caused by the browser or the JVM? > > One thing we should investigate is how many requests the UI sends to the > JobManager, > but I don't know of a good way to do that without modifying the source. > > On 06.12.2017 01:20, Yuta Morisawa wrote: >> Hi Timo >> >> I execute streaming job without checkpointing and I don't configure >> any state backend, so it may be "MemoryStateBackend". >> >> Actually, my streaming app just reads data from kafka and writes it to >> an external DB. Its not so complicated. >> >> Regards, >> Yuta >> >> On 2017/12/05 19:55, Timo Walther wrote: >>> I had some profiling tool like jvisualvm in mind. Are you executing >>> streaming or batch jobs? If streaming, is checkpointing enabled and >>> which type of statebackend? >>> >>> @Chesnay do you have experience with slow behavior of the Web UI? >>> >>> Regards, >>> Timo >>> >>> >>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: >>>> Hi Timo >>>> >>>> Thank you for your early reply. >>>> >>>> These are commands which I run my apps. >>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 >>>> ./bin/flink run -p 100 <jar1> <jar1 parameters> >>>> ./bin/flink run -p 100 <jar2> <jar2 parameters> >>>> >>>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs. >>>> >>>> > Maybe you can use a profiler and find out which component consumes so >>>> > much CPU resources? >>>> You mean Java Flight Recorder or JITWatch? >>>> Or, Flink has original profiler? >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html >>>> >>>> >>>> Regards, >>>> Yuta >>>> >>>> On 2017/12/05 18:02, Timo Walther wrote: >>>>> Hi Yuta, >>>>> >>>>> as far as I know you cannot assign more cores to a JobManager. >>>>> >>>>> Can you tell us a bit more about your environment? How many jobs >>>>> does the JobManager has to manage? How much heap memory is assigned >>>>> to the JobManager? >>>>> >>>>> Maybe you can use a profiler and find out which component consumes >>>>> so much CPU resources? >>>>> >>>>> Regards, >>>>> Timo >>>>> >>>>> >>>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >>>>>> Hi >>>>>> >>>>>> Now I am looking for the way to increase the number of allocated >>>>>> CPU cores because my JobManagaer WEBUI is very heavy and sometimes >>>>>> freeze. >>>>>> >>>>>> I think this is caused by the resource shortage of JobManager. >>>>>> How can I increase the number of CPU for JobManager in YARN mode? >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> Yuta >>>>> >>>>> >>>>> >>> >>> >> >> > > |
|
|
A 'rebalance()' should no impose load on the JM itself. There are more sub-channels in that case, but the JobManager is not coordinating anything for them unless there is a failure/recovery.
Each subchannel generates additional metrics, though - might that be an issue? On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa <[hidden email]> wrote: > Which version of Fllink are you using? |
|
|
This can only have an effect if "taskmanager.net.detailed-metrics"
is enabled (by default it is disabled).
Otherwise an increase in sub-channels will neither increase the number of metrics nor the number of polls by the UI. On 06.12.2017 10:22, Stephan Ewen wrote:
|
|
|
That said, metrics could still be the
issue, with FLINK-7368
being the prime suspect. This was fixed for 1.3.3 and 1.4 (both
unreleased though).
On 06.12.2017 10:36, Chesnay Schepler wrote:
|
|
|
Thank you for information.
I will investigate my cluster more carefully. Anyway, at this moment, the WEBUI works well(I don't know why). If I get some updates, I will inform you. By the way, I have another problem... When I run multiple Flink jobs in YARN, the job-logs are mixed. cf) yarn logs -applicationId <id> How can I avoid it? Regards, Yuta On 2017/12/06 18:52, Chesnay Schepler wrote: > That said, metrics could still be the issue, with FLINK-7368 > <https://issues.apache.org/jira/browse/FLINK-7368> being the prime > suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though). > > On 06.12.2017 10:36, Chesnay Schepler wrote: >> This can only have an effect if "*taskmanager.net.detailed-metrics*" >> is enabled (by default it is disabled). >> Otherwise an increase in sub-channels will neither increase the number >> of metrics nor the number of polls by the UI. >> >> On 06.12.2017 10:22, Stephan Ewen wrote: >>> A 'rebalance()' should no impose load on the JM itself. There are >>> more sub-channels in that case, but the JobManager is not >>> coordinating anything for them unless there is a failure/recovery. >>> >>> Each subchannel generates additional metrics, though - might that be >>> an issue? >>> >>> On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa >>> <[hidden email] <mailto:[hidden email]>> >>> wrote: >>> >>> > Which version of Fllink are you using? >>> I'm using Flink 1.3.0. >>> >>> > Is the high CPU usage caused by the browser or the JVM? >>> The CPU usage of browser is not high, the JVM is not also. >>> >>> I found that if I remove the rebalance function in my code, the >>> WEBUI works well. >>> I don't know why but I think rebalance function imposes a load on >>> JobManager. >>> >>> >>> Regards, >>> Yuta >>> >>> >>> On 2017/12/06 17:08, Chesnay Schepler wrote: >>> >>> Which version of Fllink are you using? >>> >>> Is the high CPU usage caused by the browser or the JVM? >>> >>> One thing we should investigate is how many requests the UI >>> sends to the JobManager, >>> but I don't know of a good way to do that without modifying >>> the source. >>> >>> On 06.12.2017 01 <tel:06.12.2017%2001>:20, Yuta Morisawa wrote: >>> >>> Hi Timo >>> >>> I execute streaming job without checkpointing and I don't >>> configure any state backend, so it may be >>> "MemoryStateBackend". >>> >>> Actually, my streaming app just reads data from kafka and >>> writes it to an external DB. Its not so complicated. >>> >>> Regards, >>> Yuta >>> >>> On 2017/12/05 19:55, Timo Walther wrote: >>> >>> I had some profiling tool like jvisualvm in mind. Are >>> you executing streaming or batch jobs? If streaming, >>> is checkpointing enabled and which type of statebackend? >>> >>> @Chesnay do you have experience with slow behavior of >>> the Web UI? >>> >>> Regards, >>> Timo >>> >>> >>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa: >>> >>> Hi Timo >>> >>> Thank you for your early reply. >>> >>> These are commands which I run my apps. >>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10 >>> ./bin/flink run -p 100 <jar1> <jar1 parameters> >>> ./bin/flink run -p 100 <jar2> <jar2 parameters> >>> >>> So, JobManager Heap Memory = 6000 MB and it >>> manages 2 jobs. >>> >>> > Maybe you can use a profiler and find out which >>> component consumes so >>> > much CPU resources? >>> You mean Java Flight Recorder or JITWatch? >>> Or, Flink has original profiler? >>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html >>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html> >>> >>> >>> Regards, >>> Yuta >>> >>> On 2017/12/05 18:02, Timo Walther wrote: >>> >>> Hi Yuta, >>> >>> as far as I know you cannot assign more cores >>> to a JobManager. >>> >>> Can you tell us a bit more about your >>> environment? How many jobs does the >>> JobManager has to manage? How much heap >>> memory is assigned to the JobManager? >>> >>> Maybe you can use a profiler and find out >>> which component consumes so much CPU resources? >>> >>> Regards, >>> Timo >>> >>> >>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: >>> >>> Hi >>> >>> Now I am looking for the way to increase >>> the number of allocated CPU cores because >>> my JobManagaer WEBUI is very heavy and >>> sometimes freeze. >>> >>> I think this is caused by the resource >>> shortage of JobManager. >>> How can I increase the number of CPU for >>> JobManager in YARN mode? >>> >>> >>> >>> Thanks >>> Yuta >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |