[PATCH blktests v2 00/11] support test case repeat by different conditions

[Nitesh Shetty]

On 16/04/24 07:31PM, Shin'ichiro Kawasaki wrote:
>In the recent discussion for nvme test group [1], two pain points were mentioned
>regarding the test case runs.
>
>1) Several test cases in nvme test group do exactly the same test except the
>   NVME transport backend set up condition difference (device vs. file). This
>   results in duplicate test script codes. It is desired to unify the test cases
>   and run them repeatedly with the different conditions.
>
>2) NVME transport types can be specified with nvme_trtype parameter so that the
>   same tests can be run for various transport types. However, some test cases
>   do not depend on the transport types. They are repeated in multiple runs for
>   the various transport types under the exact same conditions. It is desired to
>   repeat the test cases only when such repetition is required.
>
>[1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/
>
>One idea to address these pain points is to add the test repeat feature to the
>nvme test group. However, Daniel questioned if the feature could be implemented
>in the blktests framework. Actually, a similar feature has already been
>implemented to repeat some test cases for non-zoned block devices and zoned
>block devices. However, this feature is implemented only for the zoned and non-
>zoned device conditions. It can not fulfill the desires for nvme test group.
>
>This series proposes to generalize the feature in the blktests framework to
>repeat the test cases with different conditions. Introduce a new function
>set_conditions() that each test case can define and instruct the framework to
>repeat the test case. The first four patches introduce the feature and apply it
>to the repetition for non-zoned and zoned block devices. The following seven
>patches apply the feature to nvme test group so that the test cases can be
>repeated for NVME transport types and backend types in the ideal way. Two of the
>seven patches are reused from the work by Daniel. The all patches are listed in
>the order that does not lose the test coverage with the default set up.
>
>This series introduces new config parameters NVMET_TRTYPES and
>NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When
>they are defined in the config file as follows,
>
>  NVMET_TRTYPES="loop rdma tcp"
>  NVMET_BLKDEV_TYPES="device file"
>
>the test cases which depend on these parameters are repeated 3 x 2 = 6 times.
>For example, nvme/006 is repeated as follows.
>
>nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed]
>    runtime  0.148s  ...  0.165s
>nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed]
>    runtime  0.273s  ...  0.235s
>nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target)  [passed]
>    runtime  0.162s  ...  0.164s
>nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target)   [passed]
>    runtime  0.138s  ...  0.134s
>nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target)   [passed]
>    runtime  0.216s  ...  0.201s
>nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target)    [passed]
>    runtime  0.154s  ...  0.146s
>
>
>Changes from v1:
>* Renamed NVMET_TR_TYPES to NVMET_TRTYPES
>* 1st patch: reflected comments on the list and added Reviewed-by tag
>* 5th patch: changed NVMET_TRTYPES from array to variable
>* 7th patch: changed NVMET_BLKDEV_TYPES from array to variable
>* Reflected other comments on the list
>
>
>Daniel Wagner (3):
>  nvme/rc: add blkdev type environment variable
>  nvme/{007,009,011,013,015,020,024}: drop duplicate nvmet blkdev type
>    tests
>  nvme/{021,022,025,026,027,028}: do not hard code target blkdev type
>
>Shin'ichiro Kawasaki (8):
>  check: factor out _run_test()
>  check: support test case repeat by different conditions
>  check: use set_conditions() for the CAN_BE_ZONED test cases
>  meta/{016,017}: add test cases to check repeated test case runs
>  nvme/rc: introduce NVMET_TRTYPES
>  nvme/rc: introduce NVMET_BLKDEV_TYPES
>  nvme/{002-031,033-038,040-045,047,048}: support NMVET_TRTYPES
>  nvme/{006,008,010,012,014,019,023}: support NVMET_BLKDEV_TYPES
>

Acked-by: Nitesh Shetty <nj.shetty at samsung.com>