Automated hostapd/wpa_supplicant testing with mac80211_hwsim

[Jouni Malinen]

The mac80211_hwsim-based automated testing framework for hostapd and
wpa_supplicant is getting pretty complete and there is now some
documentation available in tests/hwsim/README to describe the setup and
to provide step-by-step instructions on how to get this running on your
own (only a single Linux device is needed).

http://w1.fi/gitweb/gitweb.cgi?p=hostap.git;a=blob_plain;f=tests/hwsim/README;hb=HEAD

There are now 75 test cases (see the list below) in the repository to
give reasonable coverage on normal functionality and also to test some
protocol level conformance. These test cases are run automatically on
every hostap.git commit and I'm moving more towards using a temporary
pending branch to check for regressions before pushing new commits into
the master branch.

Obviously, it would be even nicer if test coverage were greater and
that's something that I'm pushing towards by adding new test cases for
existing functionality and also trying to do that for all new
functionality added to hostapd and wpa_supplicant. That said, it would
certainly be welcome to receive contributions to extend the existing
test cases and add new ones, so that I'm not the only one coming up with
the tests. This can be done with simple python scripts following the
existing examples in tests/hwsim/test_*.py, so no special knowledge or
skills are needed for this. If you have favorite functionality that you
want to make sure will not regress in future releases, contributing a
test case to verify that part of the system would be a good way to do
that.. ;-)

In addition, this design provides quite convenient environment for
development purposes and I would recommend taking a look at this if you
are working on any wpa_supplicant and hostapd changes (or
cfg80211/mac80211 for that matter). Building the test cases concurrently
with new functionality makes it easy to test the changes during
development and results in have the test cases easily available for
regression testing once finished.

If you are not yet familiar with mac80211_hwsim, you may find it
surprisingly nice environment for early (and in many cases, for more or
less all) upper layer WLAN development. This is how I do most of my work
on wpa_supplicant, hostapd, cfg80211, and mac80211 nowadays and wouldn't
really want to have to survive without this type of environment that
protects me from flaky development platforms and drivers that may not
yet be as robust as one might hope. It's much more convenient to get the
upper layer changes, i.e., pretty much anything above the driver,
completed first before moving to such platforms for testing.


test_ibss_rsn - IBSS RSN
test_ap_ft - WPA2-PSK-FT AP
test_ap_ft_mixed - WPA2-PSK-FT mixed-mode AP
test_ap_ft_pmf - WPA2-PSK-FT AP with PMF
test_ap_roam_open - Roam between two open APs
test_ap_roam_wpa2_psk - Roam between two WPA2-PSK APs
test_both_go_intent_15 - P2P GO Negotiation with both devices using GO intent 15
test_both_go_neg_display - P2P GO Negotiation with both devices trying to display PIN
test_both_go_neg_enter - P2P GO Negotiation with both devices trying to enter PIN
test_grpform - P2P group formation using PIN and authorized connection (init -> GO)
test_grpform2 - P2P group formation using PIN and authorized connection (resp -> GO)
test_grpform3 - P2P group formation using PIN and re-init GO Negotiation
test_grpform_pbc - P2P group formation using PBC and re-init GO Negotiation
test_grpform_per_sta_psk - P2P group formation with per-STA PSKs
test_grpform_per_sta_psk_wps - P2P group formation with per-STA PSKs with non-P2P WPS STA
test_nfc_wps_config_token - NFC tag with configuration token from AP
test_nfc_wps_config_token_init - NFC tag with configuration token from AP with auto configuration
test_nfc_wps_handover - Connect to WPS AP with NFC connection handover
test_nfc_wps_password_token_ap - WPS registrar configuring an AP using AP password token
test_nfc_wps_password_token_sta - NFC tag with password token on the station/Enrollee
test_nfc_wps_password_token_sta_init - Initial AP configuration with first WPS NFC Enrollee
test_p2p_service_discovery - P2P service discovery
test_p2p_service_discovery_bonjour - P2P service discovery (Bonjour)
test_p2p_service_discovery_bonjour2 - P2P service discovery (Bonjour AFS)
test_p2p_service_discovery_upnp - P2P service discovery (UPnP)
test_p2p_service_discovery_upnp2 - P2P service discovery (UPnP using request helper)
test_p2p_service_discovery_ws - P2P service discovery (WS-Discovery)
test_persistent_group - P2P persistent group formation and re-invocation
test_persistent_group_invite_removed_client - P2P persistent group client removal and re-invitation
test_persistent_group_per_sta_psk - P2P persistent group formation and re-invocation using per-client PSK
test_ap_open_tdls - Open AP and two stations using TDLS
test_ap_wep_tdls - WEP AP and two stations using TDLS
test_ap_wpa2_tdls - WPA2-PSK AP and two stations using TDLS
test_ap_wpa2_tdls_concurrent_init - Concurrent TDLS setup initiation
test_ap_wpa2_tdls_concurrent_init2 - Concurrent TDLS setup initiation (reverse)
test_ap_wpa2_tdls_decline_resp - Decline TDLS Setup Response
test_ap_wpa2_tdls_diff_rsnie - TDLS with different RSN IEs
test_ap_wpa2_tdls_long_frame - TDLS with long setup/teardown frames
test_ap_wpa2_tdls_long_lifetime - TDLS with long TPK lifetime
test_ap_wpa2_tdls_reneg - Renegotiate TDLS link
test_ap_wpa2_tdls_wrong_lifetime_resp - Incorrect TPK lifetime in TDLS Setup Response
test_ap_wpa_mixed_tdls - WPA+WPA2-PSK AP and two stations using TDLS
test_ap_wpa_tdls - WPA-PSK AP and two stations using TDLS
test_autogo - P2P autonomous GO and client joining group
test_autogo_2cli - P2P autonomous GO and two clients joining group
test_autogo_legacy - P2P autonomous GO and legacy clients
test_autogo_tdls - P2P autonomous GO and two clients using TDLS
test_discovery - P2P device discovery and provision discovery
test_discovery_group_client - P2P device discovery for a client in a group
test_ap_wps_conf - WPS PBC provisioning with configured AP
test_ap_wps_conf_pin - WPS PIN provisioning with configured AP
test_ap_wps_er_add_enrollee - WPS ER configuring AP and adding a new enrollee using PIN
test_ap_wps_er_add_enrollee_pbc - WPS ER connected to AP and adding a new enrollee using PBC
test_ap_wps_init - Initial AP configuration with first WPS Enrollee
test_ap_wps_pbc_overlap_2ap - WPS PBC session overlap with two active APs
test_ap_wps_pbc_overlap_2sta - WPS PBC session overlap with two active STAs
test_ap_wps_reg_config - WPS registrar configuring and AP using AP PIN
test_ap_wps_reg_connect - WPS registrar using AP PIN to connect
test_ap_hs20_select - Hotspot 2.0 network selection
test_p2p_go_invite - P2P GO inviting a client to join
test_ap_pmf_negative - WPA2-PSK AP without PMF (negative test)
test_ap_pmf_optional - WPA2-PSK AP with PMF optional
test_ap_pmf_optional_2akm - WPA2-PSK AP with PMF optional (2 AKMs)
test_ap_pmf_required - WPA2-PSK AP with PMF required
test_ap_wpa2_eap_aka - WPA2-Enterprise connection using EAP-AKA
test_ap_wpa2_eap_aka_prime - WPA2-Enterprise connection using EAP-AKA'
test_ap_wpa2_eap_peap_eap_mschapv2 - WPA2-Enterprise connection using EAP-PEAP/EAP-MSCHAPv2
test_ap_wpa2_eap_sim - WPA2-Enterprise connection using EAP-SIM
test_ap_wpa2_eap_ttls_chap - WPA2-Enterprise connection using EAP-TTLS/CHAP
test_ap_wpa2_eap_ttls_eap_gtc - WPA2-Enterprise connection using EAP-TTLS/EAP-GTC
test_ap_wpa2_eap_ttls_eap_md5 - WPA2-Enterprise connection using EAP-TTLS/EAP-MD5
test_ap_wpa2_eap_ttls_eap_mschapv2 - WPA2-Enterprise connection using EAP-TTLS/EAP-MSCHAPv2
test_ap_wpa2_eap_ttls_mschap - WPA2-Enterprise connection using EAP-TTLS/MSCHAP
test_ap_wpa2_eap_ttls_mschapv2 - WPA2-Enterprise connection using EAP-TTLS/MSCHAPv2
test_ap_wpa2_eap_ttls_pap - WPA2-Enterprise connection using EAP-TTLS/PAP

-- 
Jouni Malinen                                            PGP id EFC895FA