On Flink job re-submission, observing error, "the rpc invocation size exceeds the maximum akka framesize"

[shravan]

Hi,

This is in continuation to an already raised request, (had replied to the
same thread but couldn't get any response yet, hence posting a new request)
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html

We are observing the same error as well with regard to "The rpc invocation
size exceeds the maximum akka framesize.", and have follow-up questions on
the same.

Why we face this issue, how can we know the expected size for which it is
failing? The error message does not indicate that. Does the operator state
have any impact on the expected Akka frame size? What is the impact of
increasing it?

Awaiting a response.

Regards,
Shravan



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

> how can we know the expected size for which it is failing?

If you did not configure akka.framesize yourself then it is set to the
documented default value. See the configuration documentation for the
release you are using.

 > Does the operator state have any impact on the expected Akka frame size?

If you are using the MemoryStateBackend, yes. Otherwise, the impact of
using any form of state on the framesize should be negligible.

 > What is the impact of increasing it?

Increase in memory consumption, probably around 1-2x the increased
amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)

On 9/18/2020 9:50 AM, shravan wrote:

> Hi,
>
> This is in continuation to an already raised request, (had replied to the
> same thread but couldn't get any response yet, hence posting a new request)
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
>
> We are observing the same error as well with regard to "The rpc invocation
> size exceeds the maximum akka framesize.", and have follow-up questions on
> the same.
>
> Why we face this issue, how can we know the expected size for which it is
> failing? The error message does not indicate that. Does the operator state
> have any impact on the expected Akka frame size? What is the impact of
> increasing it?
>
> Awaiting a response.
>
> Regards,
> Shravan
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[shravan]

Thanks for the quick response.

I might have wrongly phrased one of the questions.

/"> how can we know the expected size for which it is failing?

If you did not configure akka.framesize yourself then it is set to the
documented default value. See the configuration documentation for the
release you are using."/

We found out the default size from the configuration but we are unable to
identify the size for which it fails. Could you help out on this?

Awaiting a response.

Regards,
Shravan




Chesnay Schepler wrote

>> how can we know the expected size for which it is failing?
>
> If you did not configure akka.framesize yourself then it is set to the
> documented default value. See the configuration documentation for the
> release you are using.
>
>  > Does the operator state have any impact on the expected Akka frame
> size?
>
> If you are using the MemoryStateBackend, yes. Otherwise, the impact of
> using any form of state on the framesize should be negligible.
>
>  > What is the impact of increasing it?
>
> Increase in memory consumption, probably around 1-2x the increased
> amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)
>
> On 9/18/2020 9:50 AM, shravan wrote:
>> Hi,
>>
>> This is in continuation to an already raised request, (had replied to the
>> same thread but couldn't get any response yet, hence posting a new
>> request)
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
>>
>> We are observing the same error as well with regard to "The rpc
>> invocation
>> size exceeds the maximum akka framesize.", and have follow-up questions
>> on
>> the same.
>>
>> Why we face this issue, how can we know the expected size for which it is
>> failing? The error message does not indicate that. Does the operator
>> state
>> have any impact on the expected Akka frame size? What is the impact of
>> increasing it?
>>
>> Awaiting a response.
>>
>> Regards,
>> Shravan
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>

--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

If you use 1.10.0 or above the framesize for which it failed is part of
the exception message, see FLINK-14618.

If you are using older version, then I'm afraid there is no way to tell.

On 9/18/2020 12:11 PM, shravan wrote:

> Thanks for the quick response.
>
> I might have wrongly phrased one of the questions.
>
> /"> how can we know the expected size for which it is failing?
>
> If you did not configure akka.framesize yourself then it is set to the
> documented default value. See the configuration documentation for the
> release you are using."/
>
> We found out the default size from the configuration but we are unable to
> identify the size for which it fails. Could you help out on this?
>
> Awaiting a response.
>
> Regards,
> Shravan
>
>
>
>
> Chesnay Schepler wrote
>>> how can we know the expected size for which it is failing?
>> If you did not configure akka.framesize yourself then it is set to the
>> documented default value. See the configuration documentation for the
>> release you are using.
>>
>>   > Does the operator state have any impact on the expected Akka frame
>> size?
>>
>> If you are using the MemoryStateBackend, yes. Otherwise, the impact of
>> using any form of state on the framesize should be negligible.
>>
>>   > What is the impact of increasing it?
>>
>> Increase in memory consumption, probably around 1-2x the increased
>> amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)
>>
>> On 9/18/2020 9:50 AM, shravan wrote:
>>> Hi,
>>>
>>> This is in continuation to an already raised request, (had replied to the
>>> same thread but couldn't get any response yet, hence posting a new
>>> request)
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
>>>
>>> We are observing the same error as well with regard to "The rpc
>>> invocation
>>> size exceeds the maximum akka framesize.", and have follow-up questions
>>> on
>>> the same.
>>>
>>> Why we face this issue, how can we know the expected size for which it is
>>> failing? The error message does not indicate that. Does the operator
>>> state
>>> have any impact on the expected Akka frame size? What is the impact of
>>> increasing it?
>>>
>>> Awaiting a response.
>>>
>>> Regards,
>>> Shravan
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[shravan]

Thanks again for the quick response.

In that case, could you tell me what are the possible factors that warrant a
framesize increase? I see the official documentation and it simply states
"If Flink fails because messages exceed this limit, then you should increase
it", which isn't very convincing.

Regards,
M S Shravan
Chesnay Schepler wrote

> If you use 1.10.0 or above the framesize for which it failed is part of
> the exception message, see FLINK-14618.
>
> If you are using older version, then I'm afraid there is no way to tell.
>
> On 9/18/2020 12:11 PM, shravan wrote:
>> Thanks for the quick response.
>>
>> I might have wrongly phrased one of the questions.
>>
>> /"> how can we know the expected size for which it is failing?
>>
>> If you did not configure akka.framesize yourself then it is set to the
>> documented default value. See the configuration documentation for the
>> release you are using."/
>>
>> We found out the default size from the configuration but we are unable to
>> identify the size for which it fails. Could you help out on this?
>>
>> Awaiting a response.
>>
>> Regards,
>> Shravan
>>
>>
>>
>>
>> Chesnay Schepler wrote
>>>> how can we know the expected size for which it is failing?
>>> If you did not configure akka.framesize yourself then it is set to the
>>> documented default value. See the configuration documentation for the
>>> release you are using.
>>>
>>>   > Does the operator state have any impact on the expected Akka frame
>>> size?
>>>
>>> If you are using the MemoryStateBackend, yes. Otherwise, the impact of
>>> using any form of state on the framesize should be negligible.
>>>
>>>   > What is the impact of increasing it?
>>>
>>> Increase in memory consumption, probably around 1-2x the increased
>>> amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)
>>>
>>> On 9/18/2020 9:50 AM, shravan wrote:
>>>> Hi,
>>>>
>>>> This is in continuation to an already raised request, (had replied to
>>>> the
>>>> same thread but couldn't get any response yet, hence posting a new
>>>> request)
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
>>>>
>>>> We are observing the same error as well with regard to "The rpc
>>>> invocation
>>>> size exceeds the maximum akka framesize.", and have follow-up questions
>>>> on
>>>> the same.
>>>>
>>>> Why we face this issue, how can we know the expected size for which it
>>>> is
>>>> failing? The error message does not indicate that. Does the operator
>>>> state
>>>> have any impact on the expected Akka frame size? What is the impact of
>>>> increasing it?
>>>>
>>>> Awaiting a response.
>>>>
>>>> Regards,
>>>> Shravan
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>

--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Chesnay Schepler]

There are quite a few reason why the framesize could be exceeded.

The most common one we see is due to the parallelism being so high that
tasks can't be deployed in the first place. When a task is deployed the
RPC payload also contains information about all downstream tasks this
task sends data to; when those are a few thousand (usually in case of a
shuffle) the amount of data can quickly add up.

Other causes could be tasks having thousands of accumulators or there
being too many metrics on one TaskExecutor (which would result in
metrics not being queryable from the WebUI/REST API).

Overall though, the documentation is pretty accurate. The framesize
being exceeded is usually not because the user did anything wrong, but
just operating at a scale that the default framesize cannot support. The
only solution to that is to increase the framesize.

On 9/18/2020 12:34 PM, shravan wrote:

> Thanks again for the quick response.
>
> In that case, could you tell me what are the possible factors that warrant a
> framesize increase? I see the official documentation and it simply states
> "If Flink fails because messages exceed this limit, then you should increase
> it", which isn't very convincing.
>
> Regards,
> M S Shravan
> Chesnay Schepler wrote
>> If you use 1.10.0 or above the framesize for which it failed is part of
>> the exception message, see FLINK-14618.
>>
>> If you are using older version, then I'm afraid there is no way to tell.
>>
>> On 9/18/2020 12:11 PM, shravan wrote:
>>> Thanks for the quick response.
>>>
>>> I might have wrongly phrased one of the questions.
>>>
>>> /"> how can we know the expected size for which it is failing?
>>>
>>> If you did not configure akka.framesize yourself then it is set to the
>>> documented default value. See the configuration documentation for the
>>> release you are using."/
>>>
>>> We found out the default size from the configuration but we are unable to
>>> identify the size for which it fails. Could you help out on this?
>>>
>>> Awaiting a response.
>>>
>>> Regards,
>>> Shravan
>>>
>>>
>>>
>>>
>>> Chesnay Schepler wrote
>>>>> how can we know the expected size for which it is failing?
>>>> If you did not configure akka.framesize yourself then it is set to the
>>>> documented default value. See the configuration documentation for the
>>>> release you are using.
>>>>
>>>>    > Does the operator state have any impact on the expected Akka frame
>>>> size?
>>>>
>>>> If you are using the MemoryStateBackend, yes. Otherwise, the impact of
>>>> using any form of state on the framesize should be negligible.
>>>>
>>>>    > What is the impact of increasing it?
>>>>
>>>> Increase in memory consumption, probably around 1-2x the increased
>>>> amount. (So, increase it my 1mb, memory usages goes up by 1-2 mb)
>>>>
>>>> On 9/18/2020 9:50 AM, shravan wrote:
>>>>> Hi,
>>>>>
>>>>> This is in continuation to an already raised request, (had replied to
>>>>> the
>>>>> same thread but couldn't get any response yet, hence posting a new
>>>>> request)
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/The-rpc-invocation-size-exceeds-the-maximum-akka-framesize-when-the-job-was-re-submitted-td37507.html
>>>>>
>>>>> We are observing the same error as well with regard to "The rpc
>>>>> invocation
>>>>> size exceeds the maximum akka framesize.", and have follow-up questions
>>>>> on
>>>>> the same.
>>>>>
>>>>> Why we face this issue, how can we know the expected size for which it
>>>>> is
>>>>> failing? The error message does not indicate that. Does the operator
>>>>> state
>>>>> have any impact on the expected Akka frame size? What is the impact of
>>>>> increasing it?
>>>>>
>>>>> Awaiting a response.
>>>>>
>>>>> Regards,
>>>>> Shravan
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>>>
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>