Out of memory problem with newer firmware
chris at martin.cc
chris at martin.cc
Mon Apr 11 08:46:54 EDT 2011
2011/4/11 Rafał Miłecki <zajec5 at gmail.com>:
> W dniu 11 kwietnia 2011 11:59 użytkownik chris at martin.cc
> <chris at martin.cc> napisał:
>> 2011/4/11 Rafał Miłecki <zajec5 at gmail.com>
>>>
>>> Summary:
>>> 1) Chris has ASUS WL500pv2 with OpenWRT
>>> 2) This is router with Broadcom 5354 SoC and LP-PHY card
>>> 3) Firmware 410.2160 (broadcom-wl-4.150.10.5.tar.bz2) works OK
>>> 4) Firmware 478.104 (broadcom-wl-4.178.10.4.tar.bz2) causes OOM
>>>
>>> Larry was testing his LP-PHY with kmemleak but didn't find anything.
>>>
>>> Chris traced a lot of skbs being allocated before OOM (500+).
>>>
>>> I suspect that with new firmware we provide slightly different RX
>>> packets to mac80211 and there is sth wrong with mac80211 causing SKBs
>>> not being freed. I don't have any other idea how changing firmware
>>> could cause out of memory problems.
>>>
>>> The alloc code is question is dma.c::~585
>>> skb = __dev_alloc_skb(ring->rx_buffersize, gfp_flags);
>>>
>>> I've written patch to dump data we provide mac80211. My hope is to
>>> notice differences in that data between older vs. newer firmware.
>>>
>>> Chris: could you try attached patch and provide dmesg from older and
>>> newer firmware with this patch applied?
>>>
>>> --
>>> Rafał
>>
>>
>> I applied the patch to the latest OpenWrt trunk. (rev 26588) with
>> compat-wireless 2011-04-06
>>
>> Experimental (478.104)
>> -----------------------------------------------------------------------------------------------------
>> b43-phy0: Loading firmware version 478.104 (2008-07-01 00:50:23)
>> b43-phy0: Loading firmware version 478.104 (2008-07-01 00:50:23)
>> device wlan0 entered promiscuous mode
>> br-lan: port 2(wlan0) entering forwarding state
>> br-lan: port 2(wlan0) entering forwarding state
>> b43-phy0 ERROR: RX: phystat0:0x0000 phystat3:0x3140 macstat:0x01060000
>> chanstat:0x000D mactime:0x1FE0
>> b43-phy0 ERROR: RX: ieee80211_rx_status: band:0 freq:31 signal:15
>> antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX: phystat0:0x0000 phystat3:0x8540 macstat:0x01060000
>> chanstat:0x000D mactime:0x3EB7
>> b43-phy0 ERROR: RX: ieee80211_rx_status: band:0 freq:31 signal:-74
>> antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX: phystat0:0x0000 phystat3:0x3140 macstat:0x01060000
>> chanstat:0x000D mactime:0x779E
>> b43-phy0 ERROR: RX: ieee80211_rx_status: band:0 freq:31 signal:-2
>> antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX: phystat0:0x0000 phystat3:0x3140 macstat:0x01060000
>> chanstat:0x000D mactime:0x1D85
>> b43-phy0 ERROR: RX: ieee80211_rx_status: band:0 freq:31 signal:-14
>> antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX: phystat0:0x0000 phystat3:0x8540 macstat:0x01060000
>> chanstat:0x000D mactime:0x5142
>> b43-phy0 ERROR: RX: ieee80211_rx_status: band:0 freq:31 signal:-50
>> antenna:0 rate_idx:0 flag:0x0080
>>
>> Followed immediately by OOM
>
> It's weird you got only 5 of my error-debugging messages. Earlier you
> got ~500 allocs before OOM *and* I expected each alloc to be followed
> by my error-debugging messages.
>
>
>> Stable (410.2160)
>> -----------------------------------------------------------------------------------------------------
>> b43-phy0: Loading firmware version 410.2160 (2007-05-26 15:32:10)
>> b43-phy0: Loading firmware version 410.2160 (2007-05-26 15:32:10)
>> device wlan0 entered promiscuous mode
>> br-lan: port 2(wlan0) entering forwarding state
>> br-lan: port 2(wlan0) entering forwarding state
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x7940 macstat:0x01060000 chanstat:0x005D mactime:0xC2F6
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-19 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0020 phystat3:0x8540 macstat:0x01060004 chanstat:0x005D mactime:0xCD95
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-74 antenna:1 rate_idx:0 flag:0x0000
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x2240 macstat:0x01060000 chanstat:0x005D mactime:0x2B07
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-24 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0001 phystat3:0x31A0 macstat:0x01060000 chanstat:0x005D mactime:0xEAE0
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-19 antenna:0 rate_idx:11 flag:0x0000
>> b43-phy0 ERROR: RX:
>> phystat0:0x0001 phystat3:0x31A0 macstat:0x01060000 chanstat:0x005D mactime:0xEF34
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-22 antenna:0 rate_idx:11 flag:0x0000
>> device wlan0 left promiscuous mode
>> br-lan: port 2(wlan0) entering forwarding state
>> device wlan0 entered promiscuous mode
>> br-lan: port 2(wlan0) entering forwarding state
>> br-lan: port 2(wlan0) entering forwarding state
>> net_ratelimit: 348 callbacks suppressed
>> b43-phy0 ERROR: RX:
>> phystat0:0x0020 phystat3:0xB540 macstat:0x01068000 chanstat:0x005D mactime:0x52E8
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-21 antenna:1 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x3140 macstat:0x01068028 chanstat:0x005D mactime:0x5985
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-21 antenna:0 rate_idx:0 flag:0x0000
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x2240 macstat:0x01068000 chanstat:0x005D mactime:0xBB50
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-13 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x8540 macstat:0x01068000 chanstat:0x005D mactime:0x16DC
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-76 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x8540 macstat:0x01068000 chanstat:0x005D mactime:0x2000
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-77 antenna:0 rate_idx:0 flag:0x0080
>> net_ratelimit: 400 callbacks suppressed
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x3140 macstat:0x01068000 chanstat:0x005D mactime:0xE2DA
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-22 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0020 phystat3:0xB540 macstat:0x01068000 chanstat:0x005D mactime:0x4B9B
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-11 antenna:1 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x3140 macstat:0x01068000 chanstat:0x005D mactime:0x72D9
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-22 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x3140 macstat:0x01068000 chanstat:0x005D mactime:0xDB9C
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:2 antenna:0 rate_idx:0 flag:0x0080
>> b43-phy0 ERROR: RX:
>> phystat0:0x0000 phystat3:0x8540 macstat:0x01068000 chanstat:0x005D mactime:0x40E8
>> b43-phy0 ERROR: RX: ieee80211_rx_status:
>> band:0 freq:33 signal:-79 antenna:0 rate_idx:0 flag:0x0080
>> NOHZ: local_softirq_pending 08
>> NOHZ: local_softirq_pending 08
>> NOHZ: local_softirq_pending 08
>>
>> No More messages after this, The device functions OK
>
> Here I also expected to get 500 messages, unless there was not traffic
> in the air.
>
> Conclusions:
> 1) I need to prepare another patch with more detailed debugging
> 2) There is difference in channel between firmwares. Channel is
> (chanstat & 0x7F8) >> 3. In case of 478.104 it seems to got channel
> (0x1000>>3)==1, in case of 410.2160 it seems you got channel
> (0x51>>3)==11. Do you use AP mode? Did you switch between channels?
>
> --
> Rafał
>
I am using a more recent version of compat-wireless, with quite a few
changes in mac80211
The configuration I am testing is the standard config from the openwrt build:
* AP (Master) mode.
* Channel 11
* No Encryption
* Single SSID
* Bridged
If I take it out of the bridge, it takes a little longer to fail:
STA / AP mode doesn't appear to make any difference.
The Channel is "defiantly" set the same between firmware versions
I think that you may be on to something with that.
----------------------------------------------------------
Chris Martin
m: +61 419 812 371
----------------------------------------------------------
More information about the b43-dev
mailing list