[PATCH v2 04/11] iommu/arm-smmu-v3: Drain in-flight fault handlers

[Nicolin Chen]

When a device is switching away from a domain, either through a detach or a
replace operation, in-flight stall events for the old domain might still be
on the SMMU's hardware event queue or on the IOMMU core's IOPF queue. Thus,
if the IOMMU core swaps the device's attach_handle and frees the old domain
before those handlers complete, the IOPF work might hit use-after-free.

Two queues need to be drained: the SMMU hardware event queue and the IOMMU
core IOPF software workqueue.

Poll the evtq so that pending IRQs won't let the threaded handler run after
the drain and queue a fault against the freed master_domain.

Then, synchronize_irq() on the evtq. queue_remove_raw() advances MMIO CONS
before the caller pushes an event to the IOPF queue, so the first step does
not on its own guarantee every event has been queued. It must wait for the
IRQ handler to finish to close that gap.

Lastly, invoke iopf_queue_flush_dev() to drain the IOPF workqueue.

Fixes: cfea71aea921 ("iommu/arm-smmu-v3: Put iopf enablement in the domain attach path")
Cc: stable at vger.kernel.org # v6.16
Co-developed-by: Barak Biber <bbiber at nvidia.com>
Signed-off-by: Barak Biber <bbiber at nvidia.com>
Co-developed-by: Stefan Kaestle <skaestle at nvidia.com>
Signed-off-by: Stefan Kaestle <skaestle at nvidia.com>
Signed-off-by: Malak Marrid <mmarrid at nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc at nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4794a15f351c4..ffc9621cd2288 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3267,10 +3267,29 @@ void arm_smmu_attach_release(struct arm_smmu_attach_state *state)
 	struct arm_smmu_master_domain *master_domain = state->old_master_domain;
 	struct arm_smmu_master *master = state->master;
 
+	lockdep_assert_not_held(&arm_smmu_asid_lock);
 	iommu_group_mutex_assert(master->dev);
 
 	if (!master_domain)
 		return;
+
+	if (master_domain->using_iopf) {
+		struct arm_smmu_device *smmu = master->smmu;
+
+		/* Drain the hardware eventq */
+		if (master->stall_enabled) {
+			arm_smmu_drain_queue_for_iopf(smmu, &smmu->evtq.q);
+			/* Ensure pending events have reached the IOPF queue */
+			if (smmu->evtq.q.irq)
+				synchronize_irq(smmu->evtq.q.irq);
+		}
+		/* Pending events might be in the combined_irq handler */
+		if (smmu->combined_irq)
+			synchronize_irq(smmu->combined_irq);
+		/* Lastly, drain the IOPF queue */
+		iopf_queue_flush_dev(master->dev);
+	}
+
 	arm_smmu_disable_iopf(master, master_domain);
 	kfree(master_domain);
 	state->old_master_domain = NULL;
-- 
2.43.0