Re: CPU Cores of JobManager

Posted by Chesnay Schepler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/CPU-Cores-of-JobManager-tp17022p17049.html

Which version of Fllink are you using?

Is the high CPU usage caused by the browser or the JVM?

One thing we should investigate is how many requests the UI sends to the
JobManager,
but I don't know of a good way to do that without modifying the source.

On 06.12.2017 01:20, Yuta Morisawa wrote:

> Hi Timo
>
> I execute streaming job without checkpointing and I don't configure
> any state backend, so it may be "MemoryStateBackend".
>
> Actually, my streaming app just reads data from kafka and writes it to
> an external DB. Its not so complicated.
>
> Regards,
> Yuta
>
> On 2017/12/05 19:55, Timo Walther wrote:
>> I had some profiling tool like jvisualvm in mind. Are you executing
>> streaming or batch jobs? If streaming, is checkpointing enabled and
>> which type of statebackend?
>>
>> @Chesnay do you have experience with slow behavior of the Web UI?
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>> Hi Timo
>>>
>>> Thank you for your early reply.
>>>
>>> These are commands which I run my apps.
>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>
>>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>>
>>> > Maybe you can use a profiler and find out which component consumes so
>>> > much CPU resources?
>>> You mean Java Flight Recorder or JITWatch?
>>> Or, Flink has original profiler?
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html 
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>> On 2017/12/05 18:02, Timo Walther wrote:
>>>> Hi Yuta,
>>>>
>>>> as far as I know you cannot assign more cores to a JobManager.
>>>>
>>>> Can you tell us a bit more about your environment? How many jobs
>>>> does the JobManager has to manage? How much heap memory is assigned
>>>> to the JobManager?
>>>>
>>>> Maybe you can use a profiler and find out which component consumes
>>>> so much CPU resources?
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>>> Hi
>>>>>
>>>>> Now I am looking for the way to increase the number of allocated
>>>>> CPU cores because my JobManagaer WEBUI is very heavy and sometimes
>>>>> freeze.
>>>>>
>>>>> I think this is caused by the resource shortage of JobManager.
>>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>> Yuta
>>>>
>>>>
>>>>
>>
>>
>
>