[PATCH v2] hwsim tests: Add test cases for AP channel switch announcement

[Luca Coelho]

On Tue, 2014-01-14 at 17:05 +0200, Jouni Malinen wrote:
> On Fri, Jan 10, 2014 at 04:26:33PM +0200, Luca Coelho wrote:
> > On Sat, 2013-12-14 at 21:55 -0800, Jouni Malinen wrote:
> > > All of these test cases fail with older versions of
> > > mac80211/cfg80211/hwsim (e.g., with the version I'm running on my build
> > > server). As such, I cannot apply this yet.
> > 
> > I see.  So we should keep all the tests backward-compatible? Seems like
> > a lot of extra work, but makes sense since the kernel version you're
> > using to run the tests is independent of the hostap version...
> 
> Depends on what you mean with backward-compatible.. There is no need to
> be backwards compatible with older wpa_supplicant/hostapd versions, but
> there is need for the full setup to work with number of different kernel
> versions.

That's what I meant.  Backward-compatible with old kernels.


> This does not need to include everything, but in general, it
> would be nice to be able to at minimum report SKIP instead of FAIL if
> the kernel does not support something. In general, I think something
> went wrong with the kernel interface design or hostapd/wpa_supplicant
> implementation if that cannot be done easily since
> hostapd/wpa_supplicant themselves will obviously need to work with older
> kernel versions and it is useful helpful to know what functionality is
> available.

I totally agree. :)


> At minimum, there will be a hard blocker on the new test cases not
> failing on my test server since I try not to commit anything that breaks
> those tests. I may be convinced to update the Backports version there
> when proposing a new test case, but in general, there better be a good
> reason for this and why the test case could not be made to report SKIP
> instead.

Fair enough.  I tend to forget that these tests are mostly for hostap
and not the for the kernel itself.  They just do such a good job in
testing the kernel that I get carried away. :P


> > > Would there be any way for hostapd to detect whether CSA functionality
> > > is properly supported (I sure hope so..) and then make these test case
> > > report SKIP rather than FAIL?
> > 
> > It can be done, but AFAICT there's currently no way to get this
> > information via the control interface.  To do this, I we have to export
> > the capa.flags information via the control API.
> 
> That has already been done.. However, the current wireless-testing.git
> snapshot is still disabling CSA and if I re-enable it in cfg80211 and
> use the latest mac80211_hwsim contribution to claim support for it, some
> of these test cases fail due to kernel issues:
> 
> START ap_csa_1_switch 1/6
> PASS ap_csa_1_switch 4.358744 2014-01-14 15:02:09.299603
> START ap_csa_1_switch_count_0 2/6
> FAIL ap_csa_1_switch_count_0 3.481473 2014-01-14 15:02:12.795904
> START ap_csa_1_switch_count_1 3/6
> PASS ap_csa_1_switch_count_1 3.312774 2014-01-14 15:02:16.222982
> START ap_csa_2_switches 4/6
> PASS ap_csa_2_switches 5.610565 2014-01-14 15:02:21.848812
> START ap_csa_2_switches_count_0 5/6
> FAIL ap_csa_2_switches_count_0 3.788668 2014-01-14 15:02:25.652828
> START ap_csa_2_switches_count_1 6/6
> PASS ap_csa_2_switches_count_1 3.759073 2014-01-14 15:02:29.531521
> failed tests: ap_csa_1_switch_count_0 ap_csa_2_switches_count_0
> 
> With both failures due to this one:
> WARNING: CPU: 0 PID: 0 at net/mac80211/tx.c:2443 ieee80211_update_csa+0x221/0x2a0()

Right there is still some stuff missing in wireless-testing.  And, in
any case, even when all the latest submitted patches get into
wireless-testing, the tests with count 0 and count 1 will still fail
because the CSA action frame implementation for BSS is still missing.


> There needs to be a reasonable expectation of being able to run these
> tests with wireless-testing.git successfully before it is justifiable to
> add the test cases into hostap.git.
 
Right, I understand your point.

I was looking from a different perspective, a bit more like test-driven
development.  As I see, all the tests *are* valid and hostap's
implementation supports them all.  So the tests could be added to hostap
already.  If we did this, we would have a clearer visibility of the
status of CSA in the kernel.  The fact that the current kernel still
fails some tests is a bug in the kernel; not in testcases nor in hostap.

Anyway, I don't want to change your existing process.  And there's no
hurry to get these tests in the mainline, so we can wait. :)

--
Cheers,
Luca.