Broadcast storms - Voyage Linux, Engenius NL2611, hostap

[Edwin Whitelaw]

Please forgive if this is not a hostap issue but I have not been able to 
find any other discussion on the problem and it does seem possible that 
hostap could be involved.  If so, I would appreciate comments or 
redirection to a more appropriate forum to help resolve the issue.

I'm running Voyage Linux on WRAP boards, typically with a 5GHz backhaul 
in one slot and an Engenius NL2611 in the other set up as an AP.  
Engenius F/W is
prism2_hw_init: initialized in 374 ms
wifi0: NIC: id=0x8013 v1.0.0
wifi0: PRI: id=0x15 v1.1.1
wifi0: STA: id=0x1f v1.8.2

dpkg -l shows:
 hostap-utils     0.4.0-1

lsmod shows that hostap and hostap_pci are loaded

On occasion, the AP will suffer broadcast storms of ICMP dest 
unreachable packets from all the CB3 (also Engenius 2611 radios) clients 
as evidenced from the iptraf display of MAC addresses.  I also have a 
few older Tranzeo units with the same radio that will participate in 
these storms.  When they occur, I will see over 1000 pkts/sec hitting 
the AP, effectively resulting in DOS for the customers.  I have 11 of 
these units in use with a customer base per AP ranging from 36 down to 6 
users per AP.  One AP in particular has these storms almost weekly and 
two others have had them infrequently while the remainder have *never* 
experienced the problem.  The storms can occur under the most benign 
conditions, ie, light network load, no weather issues, etc.  A local 
power outage where most/all clients come back on line at once has been 
observed to initiate a storm.

The most heavily loaded AP has the same version as the problem child but 
itself has never had a broadcast storm.

My primary tool for observation has been iptraf such that I can easily 
see the type of packet, the source MAC, the pkts/s and also the normal 
traffic interspersed with the storm packets and subsequently recovery 
and normal behavior.


I have observed the following characteristics during my ongoing battle 
with this problem:

1. The current repair procedure is a script that creates an empty 
whitelist, kicks all associated clients off using kickmac and adds them 
back one at a time at roughly 10 sec intervals.  During this repair 
procedure, certain MACs can be more prone than others to re-initiate the 
storm.  Pausing the repair script can sometimes allow the system to 
stabilize on its own.  Not infrequently, there may be a very brief storm 
of 20 or so packets when adding the next client and then it stabilizes.

2. Reducing the total number of customers has allowed the repair process 
to generally work the first time through the 28 customers without the 
storm reoccurring.  Prior to that, it sometimes took several hours of 
flushing and re-adding the users before it would settle down and run 
correctly.

3. Iptables rules to deny packets has no effect.

4. The WRAP board and radio has been swapped out at the radio and board 
level with no change in behavior.  In all other respects, the system has 
been utterly reliable - uptime of 260 days and counting.

5. I have not seen this occur with Atheros based APs but also have some 
virtually identical units to the problem one that have never been a problem.

 From a high level view, the behavior seems like a classic resonance 
problem and could be a phenomenon unique to the particular equipment 
installed and even their respective distances from the AP thereby 
causing some micro-timing issues in transmission/reception with the AP.  
I'm at my wit's end in chasing this ghost.  :-(  I guess the next step 
would be to simply switch to an Atheros-based AP but the Engenius cards 
work well in all other respects and I'd really like to use them without 
fearing the dreaded storms.

Again, I have no "proof" that the hostap modules are involved in this 
but am honestly looking to see if anyone else has even experienced the 
problem, and best of all, identified a solution.

My thanks for your time to read this and offer any suggestions.  I've 
tried to provide sufficient information describing my system without 
burying the reader in minutiae.  If I've left out a key piece, please 
let me know and I'll submit that information.

Edwin Whitelaw





-- 
<=+=+=+==+=+=+==+=+=+=+=+=+=+=+=>
Edwin Whitelaw, P.E.
New River Valley Unwired, LLC
2200 Lonesome Dove Dr
Christiansburg, VA 24073
540-239-0318