Re: CPU Cores of JobManager

Posted by Yuta Morisawa on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/CPU-Cores-of-JobManager-tp17022p17073.html

Thank you for information.
I will investigate my cluster more carefully.

Anyway, at this moment, the WEBUI works well(I don't know why).
If I get some updates, I will inform you.


By the way, I have another problem...

When I run multiple Flink jobs in YARN, the job-logs are mixed.
cf) yarn logs -applicationId <id>

How can I avoid it?

Regards,
Yuta

On 2017/12/06 18:52, Chesnay Schepler wrote:

> That said, metrics could still be the issue, with FLINK-7368
> <https://issues.apache.org/jira/browse/FLINK-7368> being the prime
> suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though).
>
> On 06.12.2017 10:36, Chesnay Schepler wrote:
>> This can only have an effect if "*taskmanager.net.detailed-metrics*"
>> is enabled (by default it is disabled).
>> Otherwise an increase in sub-channels will neither increase the number
>> of metrics nor the number of polls by the UI.
>>
>> On 06.12.2017 10:22, Stephan Ewen wrote:
>>> A 'rebalance()' should no impose load on the JM itself. There are
>>> more sub-channels in that case, but the JobManager is not
>>> coordinating anything for them unless there is a failure/recovery.
>>>
>>> Each subchannel generates additional metrics, though - might that be
>>> an issue?
>>>
>>> On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa
>>> <[hidden email] <mailto:[hidden email]>>
>>> wrote:
>>>
>>>     > Which version of Fllink are you using?
>>>     I'm using Flink 1.3.0.
>>>
>>>     > Is the high CPU usage caused by the browser or the JVM?
>>>     The CPU usage of browser is not high, the JVM is not also.
>>>
>>>     I found that if I remove the rebalance function in my code, the
>>>     WEBUI works well.
>>>     I don't know why but I think rebalance function imposes a load on
>>>     JobManager.
>>>
>>>
>>>     Regards,
>>>     Yuta
>>>
>>>
>>>     On 2017/12/06 17:08, Chesnay Schepler wrote:
>>>
>>>         Which version of Fllink are you using?
>>>
>>>         Is the high CPU usage caused by the browser or the JVM?
>>>
>>>         One thing we should investigate is how many requests the UI
>>>         sends to the JobManager,
>>>         but I don't know of a good way to do that without modifying
>>>         the source.
>>>
>>>         On 06.12.2017 01 <tel:06.12.2017%2001>:20, Yuta Morisawa wrote:
>>>
>>>             Hi Timo
>>>
>>>             I execute streaming job without checkpointing and I don't
>>>             configure any state backend, so it may be
>>>             "MemoryStateBackend".
>>>
>>>             Actually, my streaming app just reads data from kafka and
>>>             writes it to an external DB. Its not so complicated.
>>>
>>>             Regards,
>>>             Yuta
>>>
>>>             On 2017/12/05 19:55, Timo Walther wrote:
>>>
>>>                 I had some profiling tool like jvisualvm in mind. Are
>>>                 you executing streaming or batch jobs? If streaming,
>>>                 is checkpointing enabled and which type of statebackend?
>>>
>>>                 @Chesnay do you have experience with slow behavior of
>>>                 the Web UI?
>>>
>>>                 Regards,
>>>                 Timo
>>>
>>>
>>>                 Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>>
>>>                     Hi Timo
>>>
>>>                     Thank you for your early reply.
>>>
>>>                     These are commands which I run my apps.
>>>                     ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>>                     ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>>                     ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>
>>>                     So, JobManager Heap Memory = 6000 MB and it
>>>                     manages 2 jobs.
>>>
>>>                     > Maybe you can use a profiler and find out which
>>>                     component consumes so
>>>                     > much CPU resources?
>>>                     You mean Java Flight Recorder or JITWatch?
>>>                     Or, Flink has original profiler?
>>>                     https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>>>                     <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html>
>>>
>>>
>>>                     Regards,
>>>                     Yuta
>>>
>>>                     On 2017/12/05 18:02, Timo Walther wrote:
>>>
>>>                         Hi Yuta,
>>>
>>>                         as far as I know you cannot assign more cores
>>>                         to a JobManager.
>>>
>>>                         Can you tell us a bit more about your
>>>                         environment? How many jobs does the
>>>                         JobManager has to manage? How much heap
>>>                         memory is assigned to the JobManager?
>>>
>>>                         Maybe you can use a profiler and find out
>>>                         which component consumes so much CPU resources?
>>>
>>>                         Regards,
>>>                         Timo
>>>
>>>
>>>                         Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>
>>>                             Hi
>>>
>>>                             Now I am looking for the way to increase
>>>                             the number of allocated CPU cores because
>>>                             my JobManagaer WEBUI is very heavy and
>>>                             sometimes freeze.
>>>
>>>                             I think this is caused by the resource
>>>                             shortage of JobManager.
>>>                             How can I increase the number of CPU for
>>>                             JobManager in YARN mode?
>>>
>>>
>>>
>>>                             Thanks
>>>                             Yuta
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>