[PATCH v3 00/34] iommu: Move iommu_group setup to IOMMU core code

Qian Cai cai at lca.pw
Tue Jun 30 20:40:20 EDT 2020


On Wed, Apr 29, 2020 at 03:36:38PM +0200, Joerg Roedel wrote:
> Hi,
> 
> here is the third version of this patch-set. Older versions can be found
> here:
> 
> 	v1: https://lore.kernel.org/lkml/20200407183742.4344-1-joro@8bytes.org/
> 	    (Has some more introductory text)
> 
> 	v2: https://lore.kernel.org/lkml/20200414131542.25608-1-joro@8bytes.org/
> 
> Changes v2 -> v3:
> 
> 	* Rebased v5.7-rc3
> 
> 	* Added a missing iommu_group_put() as reported by Lu Baolu.
> 
> 	* Added a patch to consolidate more initialization work in
> 	  __iommu_probe_device(), fixing a bug where no 'struct
> 	  device_iommu' was allocated in the hotplug path.
> 
> There is also a git-branch available with these patches applied:
> 
> 	https://git.kernel.org/pub/scm/linux/kernel/git/joro/linux.git/log/?h=iommu-probe-device-v3
> 
> Please review. If there are no objections I plan to put these patches
> into the IOMMU tree early next week.

Looks like this patchset introduced an use-after-free on arm-smmu-v3.

Reproduced using mlx5,

# echo 1 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs
# echo 0 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs 

The .config,
https://github.com/cailca/linux-mm/blob/master/arm64.config

Looking at the free stack,

iommu_release_device->iommu_group_remove_device

was introduced in 07/34 ("iommu: Add probe_device() and release_device()
call-backs").

[ 9426.724641][ T3356] pci 0000:0b:01.2: Removing from iommu group 3
[ 9426.731347][ T3356] ==================================================================
[ 9426.739263][ T3356] BUG: KASAN: use-after-free in __lock_acquire+0x3458/0x4440
__lock_acquire at kernel/locking/lockdep.c:4250
[ 9426.746477][ T3356] Read of size 8 at addr ffff0089df1a6f68 by task bash/3356
[ 9426.753601][ T3356]
[ 9426.755782][ T3356] CPU: 5 PID: 3356 Comm: bash Not tainted 5.8.0-rc3-next-20200630 #2
[ 9426.763687][ T3356] Hardware name: HPE Apollo 70             /C01_APACHE_MB         , BIOS L50_5.13_1.11 06/18/2019
[ 9426.774111][ T3356] Call trace:
[ 9426.777245][ T3356]  dump_backtrace+0x0/0x398
[ 9426.781593][ T3356]  show_stack+0x14/0x20
[ 9426.785596][ T3356]  dump_stack+0x140/0x1b8
[ 9426.789772][ T3356]  print_address_description.isra.12+0x54/0x4a8
[ 9426.795855][ T3356]  kasan_report+0x134/0x1b8
[ 9426.800203][ T3356]  __asan_report_load8_noabort+0x2c/0x50
[ 9426.805679][ T3356]  __lock_acquire+0x3458/0x4440
[ 9426.810373][ T3356]  lock_acquire+0x204/0xf10
[ 9426.814722][ T3356]  _raw_spin_lock_irqsave+0xf8/0x180
[ 9426.819853][ T3356]  arm_smmu_detach_dev+0xd8/0x4a0
arm_smmu_detach_dev at drivers/iommu/arm-smmu-v3.c:2776
[ 9426.824721][ T3356]  arm_smmu_release_device+0xb4/0x1c8
arm_smmu_disable_pasid at drivers/iommu/arm-smmu-v3.c:2754
(inlined by) arm_smmu_release_device at drivers/iommu/arm-smmu-v3.c:3000
[ 9426.829937][ T3356]  iommu_release_device+0xc0/0x178
iommu_release_device at drivers/iommu/iommu.c:302
[ 9426.834892][ T3356]  iommu_bus_notifier+0x118/0x160
[ 9426.839762][ T3356]  notifier_call_chain+0xa4/0x128
[ 9426.844630][ T3356]  __blocking_notifier_call_chain+0x70/0xa8
[ 9426.850367][ T3356]  blocking_notifier_call_chain+0x14/0x20
[ 9426.855929][ T3356]  device_del+0x618/0xa00
[ 9426.860105][ T3356]  pci_remove_bus_device+0x108/0x2d8
[ 9426.865233][ T3356]  pci_stop_and_remove_bus_device+0x1c/0x28
[ 9426.870972][ T3356]  pci_iov_remove_virtfn+0x228/0x368
[ 9426.876100][ T3356]  sriov_disable+0x8c/0x348
[ 9426.880447][ T3356]  pci_disable_sriov+0x5c/0x70
[ 9426.885117][ T3356]  mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core]
[ 9426.891549][ T3356]  sriov_numvfs_store+0x240/0x318
[ 9426.896417][ T3356]  dev_attr_store+0x38/0x68
[ 9426.900766][ T3356]  sysfs_kf_write+0xdc/0x128
[ 9426.905200][ T3356]  kernfs_fop_write+0x23c/0x448
[ 9426.909897][ T3356]  __vfs_write+0x54/0xe8
[ 9426.913984][ T3356]  vfs_write+0x124/0x3f0
[ 9426.918070][ T3356]  ksys_write+0xe8/0x1b8
[ 9426.922157][ T3356]  __arm64_sys_write+0x68/0x98
[ 9426.926766][ T3356]  do_el0_svc+0x124/0x220
[ 9426.930941][ T3356]  el0_sync_handler+0x260/0x408
[ 9426.935634][ T3356]  el0_sync+0x140/0x180
[ 9426.939633][ T3356]
[ 9426.941810][ T3356] Allocated by task 3356:
[ 9426.945985][ T3356]  save_stack+0x24/0x50
[ 9426.949986][ T3356]  __kasan_kmalloc.isra.13+0xc4/0xe0
[ 9426.955114][ T3356]  kasan_kmalloc+0xc/0x18
[ 9426.959288][ T3356]  kmem_cache_alloc_trace+0x1ec/0x318
[ 9426.964503][ T3356]  arm_smmu_domain_alloc+0x54/0x148
[ 9426.969545][ T3356]  iommu_group_alloc_default_domain+0xc0/0x440
[ 9426.975541][ T3356]  iommu_probe_device+0x1c0/0x308
[ 9426.980409][ T3356]  iort_iommu_configure+0x434/0x518
[ 9426.985452][ T3356]  acpi_dma_configure+0xf0/0x128
[ 9426.990235][ T3356]  pci_dma_configure+0x114/0x160
[ 9426.995017][ T3356]  really_probe+0x124/0x6d8
[ 9426.999364][ T3356]  driver_probe_device+0xc4/0x180
[ 9427.004232][ T3356]  __device_attach_driver+0x184/0x1e8
[ 9427.009447][ T3356]  bus_for_each_drv+0x114/0x1a0
[ 9427.014142][ T3356]  __device_attach+0x19c/0x2a8
[ 9427.018749][ T3356]  device_attach+0x10/0x18
[ 9427.023009][ T3356]  pci_bus_add_device+0x70/0xf8
[ 9427.027704][ T3356]  pci_iov_add_virtfn+0x7b4/0xb40
[ 9427.032571][ T3356]  sriov_enable+0x5c8/0xc30
[ 9427.036918][ T3356]  pci_enable_sriov+0x64/0x80
[ 9427.041485][ T3356]  mlx5_core_sriov_configure+0x58/0x260 [mlx5_core]
[ 9427.047917][ T3356]  sriov_numvfs_store+0x1c0/0x318
[ 9427.052784][ T3356]  dev_attr_store+0x38/0x68
[ 9427.057131][ T3356]  sysfs_kf_write+0xdc/0x128
[ 9427.061565][ T3356]  kernfs_fop_write+0x23c/0x448
[ 9427.066260][ T3356]  __vfs_write+0x54/0xe8
[ 9427.070346][ T3356]  vfs_write+0x124/0x3f0
[ 9427.074433][ T3356]  ksys_write+0xe8/0x1b8
[ 9427.078519][ T3356]  __arm64_sys_write+0x68/0x98
[ 9427.083127][ T3356]  do_el0_svc+0x124/0x220
[ 9427.087300][ T3356]  el0_sync_handler+0x260/0x408
[ 9427.091994][ T3356]  el0_sync+0x140/0x180
[ 9427.095992][ T3356]
[ 9427.098168][ T3356] Freed by task 3356:
[ 9427.101995][ T3356]  save_stack+0x24/0x50
[ 9427.105996][ T3356]  __kasan_slab_free+0x124/0x198
[ 9427.110777][ T3356]  kasan_slab_free+0x10/0x18
[ 9427.115210][ T3356]  slab_free_freelist_hook+0x110/0x298
[ 9427.120512][ T3356]  kfree+0x128/0x668
[ 9427.124252][ T3356]  arm_smmu_domain_free+0xf4/0x1a0
[ 9427.129206][ T3356]  iommu_group_release+0xec/0x160
[ 9427.134074][ T3356]  kobject_put+0xf4/0x238
[ 9427.138247][ T3356]  kobject_del+0x110/0x190
[ 9427.142507][ T3356]  kobject_put+0x1e4/0x238
[ 9427.146767][ T3356]  iommu_group_remove_device+0x394/0x938
[ 9427.152242][ T3356]  iommu_release_device+0x9c/0x178
iommu_release_device at drivers/iommu/iommu.c:300
[ 9427.157196][ T3356]  iommu_bus_notifier+0x118/0x160
[ 9427.162065][ T3356]  notifier_call_chain+0xa4/0x128
[ 9427.166934][ T3356]  __blocking_notifier_call_chain+0x70/0xa8
[ 9427.172670][ T3356]  blocking_notifier_call_chain+0x14/0x20
[ 9427.178233][ T3356]  device_del+0x618/0xa00
[ 9427.182406][ T3356]  pci_remove_bus_device+0x108/0x2d8
[ 9427.187535][ T3356]  pci_stop_and_remove_bus_device+0x1c/0x28
[ 9427.193271][ T3356]  pci_iov_remove_virtfn+0x228/0x368
[ 9427.198399][ T3356]  sriov_disable+0x8c/0x348
[ 9427.202746][ T3356]  pci_disable_sriov+0x5c/0x70
[ 9427.207398][ T3356]  mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core]
[ 9427.213830][ T3356]  sriov_numvfs_store+0x240/0x318
[ 9427.218698][ T3356]  dev_attr_store+0x38/0x68
[ 9427.223045][ T3356]  sysfs_kf_write+0xdc/0x128
[ 9427.227478][ T3356]  kernfs_fop_write+0x23c/0x448
[ 9427.232173][ T3356]  __vfs_write+0x54/0xe8
[ 9427.236259][ T3356]  vfs_write+0x124/0x3f0
[ 9427.240346][ T3356]  ksys_write+0xe8/0x1b8
[ 9427.244433][ T3356]  __arm64_sys_write+0x68/0x98
[ 9427.249041][ T3356]  do_el0_svc+0x124/0x220
[ 9427.253215][ T3356]  el0_sync_handler+0x260/0x408
[ 9427.257908][ T3356]  el0_sync+0x140/0x180
[ 9427.261907][ T3356]
[ 9427.264084][ T3356] The buggy address belongs to the object at ffff0089df1a6e00
[ 9427.264084][ T3356]  which belongs to the cache kmalloc-512 of size 512
[ 9427.277980][ T3356] The buggy address is located 360 bytes inside of
[ 9427.277980][ T3356]  512-byte region [ffff0089df1a6e00, ffff0089df1a7000)
[ 9427.291094][ T3356] The buggy address belongs to the page:
[ 9427.296571][ T3356] page:ffffffe02257c680 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff0089df1a1400
[ 9427.306823][ T3356] flags: 0x7ffff800000200(slab)
[ 9427.311520][ T3356] raw: 007ffff800000200 ffffffe02246b8c8 ffffffe02257ff88 ffff000000320680
[ 9427.319949][ T3356] raw: ffff0089df1a1400 00000000002a000e 00000001ffffffff ffff0089df1a5001
[ 9427.328374][ T3356] page dumped because: kasan: bad access detected
[ 9427.334630][ T3356] page->mem_cgroup:ffff0089df1a5001
[ 9427.339670][ T3356]
[ 9427.341846][ T3356] Memory state around the buggy address:
[ 9427.347322][ T3356]  ffff0089df1a6e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.355228][ T3356]  ffff0089df1a6e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.363133][ T3356] >ffff0089df1a6f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.371038][ T3356]                                                           ^
[ 9427.378337][ T3356]  ffff0089df1a6f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.386242][ T3356]  ffff0089df1a7000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 9427.394146][ T3356] ==================================================================
[ 9427.402052][ T3356] Disabling lock debugging due to kernel taint

> 
> Thanks,
> 
> 	Joerg
> 
> Joerg Roedel (33):
>   iommu: Move default domain allocation to separate function
>   iommu/amd: Implement iommu_ops->def_domain_type call-back
>   iommu/vt-d: Wire up iommu_ops->def_domain_type
>   iommu/amd: Remove dma_mask check from check_device()
>   iommu/amd: Return -ENODEV in add_device when device is not handled by
>     IOMMU
>   iommu: Add probe_device() and release_device() call-backs
>   iommu: Move default domain allocation to iommu_probe_device()
>   iommu: Keep a list of allocated groups in __iommu_probe_device()
>   iommu: Move new probe_device path to separate function
>   iommu: Split off default domain allocation from group assignment
>   iommu: Move iommu_group_create_direct_mappings() out of
>     iommu_group_add_device()
>   iommu: Export bus_iommu_probe() and make is safe for re-probing
>   iommu/amd: Remove dev_data->passthrough
>   iommu/amd: Convert to probe/release_device() call-backs
>   iommu/vt-d: Convert to probe/release_device() call-backs
>   iommu/arm-smmu: Convert to probe/release_device() call-backs
>   iommu/pamu: Convert to probe/release_device() call-backs
>   iommu/s390: Convert to probe/release_device() call-backs
>   iommu/virtio: Convert to probe/release_device() call-backs
>   iommu/msm: Convert to probe/release_device() call-backs
>   iommu/mediatek: Convert to probe/release_device() call-backs
>   iommu/mediatek-v1 Convert to probe/release_device() call-backs
>   iommu/qcom: Convert to probe/release_device() call-backs
>   iommu/rockchip: Convert to probe/release_device() call-backs
>   iommu/tegra: Convert to probe/release_device() call-backs
>   iommu/renesas: Convert to probe/release_device() call-backs
>   iommu/omap: Remove orphan_dev tracking
>   iommu/omap: Convert to probe/release_device() call-backs
>   iommu/exynos: Use first SYSMMU in controllers list for IOMMU core
>   iommu/exynos: Convert to probe/release_device() call-backs
>   iommu: Remove add_device()/remove_device() code-paths
>   iommu: Move more initialization to __iommu_probe_device()
>   iommu: Unexport iommu_group_get_for_dev()
> 
> Sai Praneeth Prakhya (1):
>   iommu: Add def_domain_type() callback in iommu_ops
> 
>  drivers/iommu/amd_iommu.c       |  97 ++++----
>  drivers/iommu/amd_iommu_types.h |   1 -
>  drivers/iommu/arm-smmu-v3.c     |  38 +---
>  drivers/iommu/arm-smmu.c        |  39 ++--
>  drivers/iommu/exynos-iommu.c    |  24 +-
>  drivers/iommu/fsl_pamu_domain.c |  22 +-
>  drivers/iommu/intel-iommu.c     |  68 +-----
>  drivers/iommu/iommu.c           | 387 +++++++++++++++++++++++++-------
>  drivers/iommu/ipmmu-vmsa.c      |  60 ++---
>  drivers/iommu/msm_iommu.c       |  34 +--
>  drivers/iommu/mtk_iommu.c       |  24 +-
>  drivers/iommu/mtk_iommu_v1.c    |  50 ++---
>  drivers/iommu/omap-iommu.c      |  99 ++------
>  drivers/iommu/qcom_iommu.c      |  24 +-
>  drivers/iommu/rockchip-iommu.c  |  26 +--
>  drivers/iommu/s390-iommu.c      |  22 +-
>  drivers/iommu/tegra-gart.c      |  24 +-
>  drivers/iommu/tegra-smmu.c      |  31 +--
>  drivers/iommu/virtio-iommu.c    |  41 +---
>  include/linux/iommu.h           |  21 +-
>  20 files changed, 531 insertions(+), 601 deletions(-)
> 
> -- 
> 2.17.1
> 
> _______________________________________________
> iommu mailing list
> iommu at lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu



More information about the Linux-rockchip mailing list