SDIO Performance once again

[Johan Adolfsson]

----- Original Message ----- 
From: "Jeff Sutherland" <jeffs at fomsystems.com>
To: "Dan Williams" <dcbw at redhat.com>
Cc: "Dominik S. Herwald" <lists at dsh-elektronik.de>; "Sven Neumann" 
<s.neumann at phase-zero.de>; <libertas-dev at lists.infradead.org>
Sent: Monday, February 09, 2009 5:30 PM
Subject: Re: SDIO Performance once again


> On Monday 09 February 2009, Dan Williams wrote:
>> On Mon, 2009-02-09 at 08:21 -0500, Jeff Sutherland wrote:
>> > On Monday 09 February 2009, Sven Neumann wrote:
>> > > Hi,
>> > >
>> > > On Sun, 2009-02-08 at 17:56 +0100, Dominik S. Herwald wrote:
>> > > > right now I am testing Marvell 8686 based Modules connected to the
>> > > > SDIO Controller of a Blackfin BF548.
>> > > >
>> > > > Basically the libertas driver works just fine and stable.
>> > > > But the Performance... :-/
>> > >
>> > > Interesting. We are having the contrary experience here. Running a
>> > > CM-X300 board, which is a PXA 300 featuring a Marvell 8686 module
>> > > connected to the SDIO controller, we are seeing a performance of 
>> > > about
>> > > 13 Mbits/sec.
>> > >
>> > > Unfortunately this is not stable. Sometimes (rarely on one of the two
>> > > boards we have, frequently on the other), there are errors:
>> > >
>> > > libertas: tx watch dog timeout
>> > >
>> > > When this happens it usually takes about six seconds for the module 
>> > > to
>> > > recover. During this period nothing is transmitted. At some point,
>> > > sooner or later, the following errors shows up:
>> > >
>> > > libertas: command 0x001f timed out
>> > > libertas: requeueing command 0x001f due to timeout (#1)
>> > > libertas: command 0x001f timed out
>> > > libertas: requeueing command 0x001f due to timeout (#2)
>> > > libertas: command 0x001f timed out
>> > > libertas: requeueing command 0x001f due to timeout (#3)
>> > > libertas: tx watch dog timeout
>> > > libertas: command 0x001f timed out
>> > > libertas: Excessive timeouts submitting command 0x001f
>> > >
>> > > At this point the module stops to work completely and I haven't found 
>> > > a
>> > > way to recover from this. The system then needs to be restarted.
>> > >
>> > > We are using linux-2.6.29-rc4 currently, but observed the same 
>> > > problems
>> > > with linux-2.6.26. The firmware version is 8.73.7p3. I have also 
>> > > tried
>> > > the 9.70.3 firmware, that didn't help either.
>> > >
>> > > Has anyone experienced similar problems? Any ideas what I could try 
>> > > to
>> > > get some more debug output from the driver?
>> > >
>> > >
>> > > Sven
>> >
>> > This looks to be caused by the same situation that affects my pxa270
>> > design with an SDIO connected 8686 chipset.  Because of the
>> > multi-threading in the driver, depending on cpu load, commands to the
>> > chip are sent in an indeterminate sequence.  If you go turn on some
>> > debugging in the driver, such
>>
>> The firmware can only execute one command at a time, so all commands are
>> stuffed into the internal command queue.  Since some commands cannot be
>> sent during PS periods, there needs to be some authority to handle that.
>> Much of this general structure is left over from the 2006 Marvell vendor
>> driver dump that Libertas was derived from.
>>
>> > as putting a printk someplace, that is usually enough of a load to pace
>> > thread execution so the commands usually execute in the same sequence 
>> > and
>> > you'll see this error go away (along with the network performance :-(
>> > Basically the 8686 firmware gets its knickers in a twist if it sees
>> > certain commands executed out of sequence.  And rightly so.  There's 
>> > some
>> > fundamental redesign of the driver that needs doing that's beyond my
>> > programming interests and abilities.
>>
>> I'm not aware of any specific situations (besides WPA setup) where
>> commands need to be executed in a specific sequence; if there were, we
>> can certainly handle that in the driver.
>>
>> Other than that, any commands that are submitted are done so at the
>> request of the user via 'iwconfig' or wpa_supplicant; these are queued
>> atomically into the main command queue.
>>
>> Much of the association process (including the worker and the back-off
>> timer stuff in assoc.c) was necessitated by the broken design of WEXT;
>> with cfg80211 I expect we can go back to a much simpler model whereby
>> the entire association request is simply sent to the firmware in one
>> shot.  There could be some cleanup opportunities here; ideally the
>> *only* communication with the firmware happens when finally setting the
>> parameters instead of the current behavior of some WEXT calls asking for
>> values before queuing up the attributes to set later.
>>
>> What sort of suggestions would you have for redesign?
>>
>> Dan
>
> Well, like I said earlier, that's really beyond my expertise as I'm a HW
> person, but when I was doing the debugging of my problems, I couldn't help
> but notice that when I was 'printk'ing' various bits and pieces of the 
> driver
> that the command sequence was always the same, regardless of CPU loading, 
> and
> that the module always worked.  Without the printk's but with driver
> debugging still turned on, I noticed that the command sequence would be
> different for the times when the firmware wedged.  So my only suggestions 
> are
> ensure that commands are not sent to the module until the previous command
> has finished executing, and maybe ensure that the sequence of commands is
> always preserved somehow.
>
> -Jeff
> -- 
> FOM Systems Inc.   www.fomsystems.com
> phone:  (USA) +1-330-628-2102 or 800-936-0561, (UK) +44-161-408-3072
> mobile: (USA) +1-330-802-1364, (World) +44-7793-827386, Skype: Adekguru
> =+=+=+=+=+=+=+=+=+=+=+=+=+=
> And He saith unto them, "Follow me, and I will make you fishers of men."
> (Matthew 4:19)

Are you using DMA?
Is the DMA cache-coherent on your platform or do you need to manually flush
or invalidate the cache in the SDIO controller code?
Are those flushes and invalidations in place if needed?

/Johan