Firmware crash when sending large numbers of forwarded packets

[Avery Pennarun]

On Fri, Mar 7, 2014 at 7:08 PM, Ben Greear <greearb at candelatech.com> wrote:
> On 03/07/2014 04:54 PM, Avery Pennarun wrote:
>> On Fri, Mar 7, 2014 at 10:53 AM, Ben Greear <greearb at candelatech.com> wrote:
>>> That is interesting, sending from LAN will often burst higher
>>> and cause more packet loss (and perhaps higher periodic packet
>>> loads) but sending locally will typically allow the local stack to back
>>> off more gracefully.
>>
>> That theory sounds right.  Do you think there's a way to simulate the
>> burstiness while avoiding the extra machine?  That would make testing
>> easier.  Some way to write a lot of packets to a socket and then
>> release it all at once?  I'm guessing UDP would work better for this
>> than TCP.
>
> UDP has some local backpressure as well, but maybe it would be more
> likely to hit it than TCP I guess.
>
> It could also be something else related to forwarding that is not just
> rate specific (maybe frames come in with LRO and something special
> about transmitting those, or something of that nature?).

Yes, that's possible...

> Are you using bridging or routing on your AP?  Bridging might
> be fundamentally different than routing as far as this bug
> is concerned.  Sending from local AP would likely emulate
> routing type behaviour.

The wlan1 interface is in a bridge, but the packets were coming from a
wired ethernet interface (WAN port) that is not in the bridge.  I'll
try without bridging and see if that affects anything.  Thanks for the
suggestion.

>>> This appears to be an assert in firmware, not just random crash.
>>>
>>> Someone with .467 firmware source and a bit of skill with the tool-chain
>>> should be able to figure out where it is asserting.
>>>
>>> I do not know anyone with both of those things, however :P
>>
>> Maybe I should try harder to get access to the firmware source :)
>
> Good luck..if you do get it and the toolchain, I can point
> you towards how to decode (ie, get stack trace) that crash, or at least I can send
> QCA email they could forward you (to satisfy any NDA isues).

Thanks.  I'll contact them again and see what happens.

Meanwhile, any suggestions anyone can provide on "what to trace" (and
how to trace) right now given that I don't have firmware access, would
be helpful.

Have fun,

Avery