[RFC v4] drm: Add initial ci/ subdirectory
Tomeu Vizoso
tomeu.vizoso at collabora.com
Tue May 17 02:24:17 PDT 2022
On 5/17/22 11:18 AM, Neil Armstrong wrote:
> On 17/05/2022 10:16, Tomeu Vizoso wrote:
>> And use it to store expectations about what the DRM drivers are
>> supposed to pass in the IGT test suite.
>>
>> Also include a configuration file that points to the out-of-tree CI
>> scripts.
>>
>> By storing the test expectations along the code we can make sure both
>> stay in sync with each other, and so we can know when a code change
>> breaks those expectations.
>>
>> This will allow all contributors to drm to reuse the infrastructure
>> already in gitlab.freedesktop.org to test the driver on several
>> generations of the hardware.
>>
>> v2:
>> - Fix names of result expectation files to match SoC
>> - Don't execute tests that are going to skip on all boards
>>
>> v3:
>> - Remove tracking of dmesg output during test execution
>>
>> v4:
>> - Move up to drivers/gpu/drm
>> - Add support for a bunch of other drivers
>> - Explain how to incorporate fixes for CI from a
>> ${TARGET_BRANCH}-external-fixes branch
>> - Remove tests that pass from expected results file, to reduce the
>> size of in-tree files
>> - Add docs about how to deal with outages in automated testing labs
>> - Specify the exact SHA of the CI scripts to be used
>>
>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso at collabora.com>
>> ---
>> Documentation/gpu/automated_testing.rst | 84 +++++++
>> drivers/gpu/drm/ci/amdgpu.testlist | 130 +++++++++++
>> drivers/gpu/drm/ci/amdgpu_stoney_results.txt | 17 ++
>> drivers/gpu/drm/ci/gitlab-ci.yml | 13 ++
>> drivers/gpu/drm/ci/i915.testlist | 219 ++++++++++++++++++
>> drivers/gpu/drm/ci/i915_amly_results.txt | 9 +
>> drivers/gpu/drm/ci/i915_apl_results.txt | 9 +
>> drivers/gpu/drm/ci/i915_cml_results.txt | 10 +
>> drivers/gpu/drm/ci/i915_glk_results.txt | 13 ++
>> drivers/gpu/drm/ci/i915_kbl_results.txt | 9 +
>> drivers/gpu/drm/ci/i915_tgl_results.txt | 12 +
>> drivers/gpu/drm/ci/i915_whl_results.txt | 9 +
>> drivers/gpu/drm/ci/mediatek.testlist | 102 ++++++++
>> .../gpu/drm/ci/mediatek_mt8173_results.txt | 11 +
>> .../gpu/drm/ci/mediatek_mt8183_results.txt | 11 +
>> .../gpu/drm/ci/mediatek_mt8192_results.txt | 11 +
>> drivers/gpu/drm/ci/meson.testlist | 71 ++++++
>> drivers/gpu/drm/ci/meson_g12b_results.txt | 11 +
>> drivers/gpu/drm/ci/msm.testlist | 144 ++++++++++++
>> drivers/gpu/drm/ci/msm_apq8016_results.txt | 77 ++++++
>> drivers/gpu/drm/ci/msm_apq8096_results.txt | 76 ++++++
>> drivers/gpu/drm/ci/msm_sc7180_results.txt | 18 ++
>> drivers/gpu/drm/ci/msm_sdm845_results.txt | 33 +++
>> drivers/gpu/drm/ci/rockchip.testlist | 188 +++++++++++++++
>> .../gpu/drm/ci/rockchip_rk3288_results.txt | 11 +
>> .../gpu/drm/ci/rockchip_rk3399_results.txt | 14 ++
>> drivers/gpu/drm/ci/virtio_gpu.testlist | 92 ++++++++
>> drivers/gpu/drm/ci/virtio_gpu_results.txt | 28 +++
>> 28 files changed, 1432 insertions(+)
>> create mode 100644 Documentation/gpu/automated_testing.rst
>> create mode 100644 drivers/gpu/drm/ci/amdgpu.testlist
>> create mode 100644 drivers/gpu/drm/ci/amdgpu_stoney_results.txt
>> create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
>> create mode 100644 drivers/gpu/drm/ci/i915.testlist
>> create mode 100644 drivers/gpu/drm/ci/i915_amly_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_apl_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_cml_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_glk_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_kbl_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_tgl_results.txt
>> create mode 100644 drivers/gpu/drm/ci/i915_whl_results.txt
>> create mode 100644 drivers/gpu/drm/ci/mediatek.testlist
>> create mode 100644 drivers/gpu/drm/ci/mediatek_mt8173_results.txt
>> create mode 100644 drivers/gpu/drm/ci/mediatek_mt8183_results.txt
>> create mode 100644 drivers/gpu/drm/ci/mediatek_mt8192_results.txt
>> create mode 100644 drivers/gpu/drm/ci/meson.testlist
>> create mode 100644 drivers/gpu/drm/ci/meson_g12b_results.txt
>> create mode 100644 drivers/gpu/drm/ci/msm.testlist
>> create mode 100644 drivers/gpu/drm/ci/msm_apq8016_results.txt
>> create mode 100644 drivers/gpu/drm/ci/msm_apq8096_results.txt
>> create mode 100644 drivers/gpu/drm/ci/msm_sc7180_results.txt
>> create mode 100644 drivers/gpu/drm/ci/msm_sdm845_results.txt
>> create mode 100644 drivers/gpu/drm/ci/rockchip.testlist
>> create mode 100644 drivers/gpu/drm/ci/rockchip_rk3288_results.txt
>> create mode 100644 drivers/gpu/drm/ci/rockchip_rk3399_results.txt
>> create mode 100644 drivers/gpu/drm/ci/virtio_gpu.testlist
>> create mode 100644 drivers/gpu/drm/ci/virtio_gpu_results.txt
>>
>> diff --git a/Documentation/gpu/automated_testing.rst
>> b/Documentation/gpu/automated_testing.rst
>> new file mode 100644
>> index 000000000000..61003ecc7b6e
>> --- /dev/null
>> +++ b/Documentation/gpu/automated_testing.rst
>> @@ -0,0 +1,84 @@
>> +.. SPDX-License-Identifier: GPL-2.0+
>> +
>> +=========================================
>> +Automated testing of the DRM subsystem
>> +=========================================
>> +
>> +
>> +Introduction
>> +============
>> +
>> +Making sure that changes to the core or drivers don't introduce
>> regressions can be very time consuming when lots of different hardware
>> configurations need to be tested. Moreover, it isn't practical for
>> each person interested in this testing to have to acquire and maintain
>> what can be a considerable amount of hardware.
>> +
>> +Also, it is desirable for developers to check for regressions in
>> their code by themselves, instead of relying on the maintainers
>> finding them and then reporting back.
>> +
>> +There are facilities in gitlab.freedesktop.org to automatically test
>> Mesa that can be used as well for testing the DRM subsystem. This
>> document explains how people interested in testing it can use this
>> shared infrastructure to save quite some time and effort.
>> +
>> +
>> +Relevant files
>> +==============
>> +
>> +drivers/gpu/drm/ci/gitlab-ci.yml
>> +--------------------------------
>> +
>> +Specifies the specific version of the scripts to be used. GitLab CI
>> will use the values defined in this file to fetch the right scripts.
>> +
>> +
>> +drivers/gpu/drm/ci/${DRIVER_NAME}.testlist
>> +------------------------------------------
>> +
>> +Specifies the tests that the current code is expected to be able to
>> reliably run for ${DRIVER_NAME}. These tests are expected to not hang
>> the DUT (Device Under Testing) when running on the revision they
>> belong to, and to give consistent results.
>> +
>> +
>> +drivers/gpu/drm/ci/${DRIVER_NAME}_*_results.txt
>> +----------------------------------------
>> +
>> +Specifies the expected results of running this specific kernel
>> revision on a given hardware configuration.
>> +
>> +
>> +How to enable automated testing on your tree
>> +============================================
>> +
>> +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you
>> don't have one yet
>> +
>> +2. In your kernel repo's configuration (eg.
>> https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change
>> the CI/CD configuration file from .gitlab-ci.yml to
>> drivers/gpu/drm/ci/gitlab-ci.yml.
>> +
>> +3. Next time you push to this repository, you will see a CI pipeline
>> being created (eg.
>> https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
>> +
>> +4. The various jobs will be run and when the pipeline is finished,
>> all jobs should be green unless a regression has been found.
>> +
>> +
>> +How to update test expectations
>> +===============================
>> +
>> +If your changes to the code fix any tests, you will have to update
>> one or more of the files in
>> drivers/gpu/drm/ci/${DRIVER_NAME}_*_results.txt, for each of the test
>> platforms affected by the change.
>> +
>> +If you have run a pipeline and it failed because of a mismatched test
>> result, you will have been offered a patch at the end of the run that
>> you can apply after reviewing it.
>> +
>> +
>> +How to expand coverage
>> +======================
>> +
>> +If your code changes makes it possible to run more tests (by solving
>> reliability issues, for example), more tests can be added to this
>> list, and then the expected results updated with the patch file that
>> will be printed at the end of each job.
>> +
>> +If there is a need for updating the version of IGT being used (maybe
>> you have added more tests to it), follow the instructions at
>> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/main#upreving-igt.
>> +
>> +
>> +How to test your changes to the scripts
>> +==========================================
>> +
>> +For testing changes to the scripts in the drm-ci repo, change the
>> DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/ci/gitlab-ci.yml to
>> match your fork of the project (eg. tomeu/drm-ci). This fork needs to
>> be in https://gitlab.freedesktop.org/.
>> +
>> +
>> +How to incorporate external fixes in your testing
>> +=================================================
>> +
>> +Often, regressions in other trees will prevent testing changes local
>> to the tree under test. These fixes will be automatically merged in
>> during the build jobs from a branch in the target tree that is named
>> as ${TARGET_BRANCH}-external-fixes.
>> +
>> +If the pipeline is not in a merge request and a branch with the same
>> name exists in the local tree, commits from that branch will be merged
>> in as well.
>> +
>> +
>> +How to deal with automated testing labs that may be down
>> +========================================================
>> +
>> +If a hardware farm is down and thus causing pipelines to fail that
>> would otherwise pass, one can disable all jobs that would be submitted
>> to that farm by editing the file at
>> https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
>>
>
>
> [..]
>
>> diff --git a/drivers/gpu/drm/ci/meson.testlist
>> b/drivers/gpu/drm/ci/meson.testlist
>> new file mode 100644
>> index 000000000000..23744b3a2fb0
>> --- /dev/null
>> +++ b/drivers/gpu/drm/ci/meson.testlist
>> @@ -0,0 +1,71 @@
>> +# Keep alphabetically sorted by default
>> +
>> +igt at core_auth@getclient-simple
>> +igt at core_auth@getclient-master-drop
>> +igt at core_auth@basic-auth
>> +igt at core_auth@many-magics
>> +igt at core_getclient
>> +igt at core_getstats
>> +igt at core_getversion
>> +igt at core_setmaster_vs_auth
>> +igt at drm_read@invalid-buffer
>> +igt at kms_addfb_basic@unused-handle
>> +igt at kms_addfb_basic@unused-pitches
>> +igt at kms_addfb_basic@unused-offsets
>> +igt at kms_addfb_basic@unused-modifier
>> +igt at kms_addfb_basic@legacy-format
>> +igt at kms_addfb_basic@no-handle
>> +igt at kms_addfb_basic@basic
>> +igt at kms_addfb_basic@bad-pitch-0
>> +igt at kms_addfb_basic@bad-pitch-32
>> +igt at kms_addfb_basic@bad-pitch-63
>> +igt at kms_addfb_basic@bad-pitch-128
>> +igt at kms_addfb_basic@bad-pitch-256
>> +igt at kms_addfb_basic@bad-pitch-1024
>> +igt at kms_addfb_basic@bad-pitch-999
>> +igt at kms_addfb_basic@bad-pitch-65536
>> +igt at kms_addfb_basic@size-max
>> +igt at kms_addfb_basic@too-wide
>> +igt at kms_addfb_basic@too-high
>> +igt at kms_addfb_basic@bo-too-small
>> +igt at kms_addfb_basic@small-bo
>> +igt at kms_addfb_basic@addfb25-modifier-no-flag
>> +igt at kms_addfb_basic@addfb25-bad-modifier
>> +igt at kms_addfb_basic@invalid-get-prop-any
>> +igt at kms_addfb_basic@invalid-get-prop
>> +igt at kms_addfb_basic@invalid-set-prop-any
>> +igt at kms_addfb_basic@invalid-set-prop
>> +igt at kms_addfb_basic@master-rmfb
>> +igt at kms_atomic_interruptible@legacy-setmode at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@atomic-setmode at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@legacy-dpms at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@legacy-pageflip at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@legacy-cursor at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@universal-setplane-primary at pipe-a-edp-1
>> +igt at kms_atomic_interruptible@universal-setplane-cursor at pipe-a-edp-1
>> +igt at kms_content_protection@LIC
>> +igt at kms_getfb@getfb-handle-zero
>> +igt at kms_getfb@getfb-handle-valid
>> +igt at kms_getfb@getfb-handle-closed
>> +igt at kms_getfb@getfb-handle-not-fb
>> +igt at kms_getfb@getfb-addfb-different-handles
>> +igt at kms_getfb@getfb-repeated-different-handles
>> +igt at kms_getfb@getfb2-handle-zero
>> +igt at kms_getfb@getfb2-handle-closed
>> +igt at kms_getfb@getfb2-handle-not-fb
>> +igt at kms_getfb@getfb2-into-addfb2
>> +igt at kms_getfb@getfb-handle-protection
>> +igt at kms_getfb@getfb2-handle-protection
>> +igt at kms_hdmi_inject@inject-4k
>> +igt at kms_prop_blob@basic
>> +igt at kms_prop_blob@blob-prop-core
>> +igt at kms_prop_blob@blob-prop-validate
>> +igt at kms_prop_blob@blob-prop-lifetime
>> +igt at kms_prop_blob@blob-multiple
>> +igt at kms_prop_blob@invalid-get-prop-any
>> +igt at kms_prop_blob@invalid-get-prop
>> +igt at kms_prop_blob@invalid-set-prop-any
>> +igt at kms_prop_blob@invalid-set-prop
>> +igt at kms_setmode@basic at pipe-a-edp-1
>> +igt at kms_setmode@basic at pipe-b-edp-1
>> +igt at kms_sysfs_edid_timing
>> diff --git a/drivers/gpu/drm/ci/meson_g12b_results.txt
>> b/drivers/gpu/drm/ci/meson_g12b_results.txt
>> new file mode 100644
>> index 000000000000..6997a75aa1ef
>> --- /dev/null
>> +++ b/drivers/gpu/drm/ci/meson_g12b_results.txt
>> @@ -0,0 +1,11 @@
>> +igt at drm_read@invalid-buffer,fail
>> +igt at kms_addfb_basic@addfb25-bad-modifier,fail
>> +igt at kms_atomic_interruptible@legacy-setmode,skip
>> +igt at kms_atomic_interruptible@atomic-setmode,skip
>> +igt at kms_atomic_interruptible@legacy-dpms,skip
>> +igt at kms_atomic_interruptible@legacy-pageflip,skip
>> +igt at kms_atomic_interruptible@legacy-cursor,skip
>> +igt at kms_atomic_interruptible@universal-setplane-primary,skip
>> +igt at kms_atomic_interruptible@universal-setplane-cursor,skip
>> +igt at kms_content_protection@lic,skip
>> +igt at kms_setmode@basic,skip
>
> Can't those be removed from testlist instead ?
> I suppose those should not skip, right ?
You are right, these slipped in. The only skips in the results should be
for tests that do run on other boards for the same driver.
> Anyway, this is great, please take my
> Reviewed-by: Neil Armstrong <narmstrong at baylibre.com>
> for stuff related to the meson driver.
Thanks,
Tomeu
More information about the linux-arm-kernel
mailing list