[PATCH] PCI: brcmstb: Reserve only the MSI vectors that are handed out
Han / 한상우Sangwoo
sangwoo.han at nearthlab.com
Fri Oct 30 05:59:10 PDT 2026
Hi Thomas,
On Thu, Oct 01 2026 at 21:29, Thomas Gleixner wrote:
> Good. Can you please retest with the updated patch?
[...]
> Any updates on that? Also please provide the source for that test.
The 5-vector endpoint I used before is no longer available to me, so I
tested on the same BCM2712 board with a similar endpoint that
advertises a single MSI vector. Since I did not have time to write a
test driver for it myself, I used an AI-generated virtual driver to
make the kernel see Multiple Message Capable = 4 and request 3 vectors.
I cannot vouch for its quality, as I had little time to review it.
Kernels: rpi-6.12.y (6.12.93) with KASAN, built with no patch, your
first patch (v1) and the updated one (v2). I ran five loads with
PCI_IRQ_AFFINITY and five without, both with the port's MSIs on the
brcmstb MSI controller and on the BCM2712 MIP, which is an MSI parent
on this tree:
no patch v1 v2
brcmstb MSI controller
KASAN report no yes yes
allocs OK, affinity 5/5 1/5 2/5
allocs OK, no affinity 2/5 (*) 5/5 5/5
free calls to the parent 3 x nr=1 4 x nr=1 4 x nr=1
BCM2712 MIP
KASAN report (**) yes yes
allocs OK, affinity 1/5 2/5
allocs OK, no affinity 5/5 5/5
free calls to the parent 4 x nr=1 1 x nr=4
(*) the original brcmstb leak
(**) not run
BUG: KASAN: slab-out-of-bounds in __irq_alloc_descs+0x158/0x460
Read of size 8 at addr ffff8000053a68b0 by task insmod/599
...
Allocated by task 599:
...
kmemdup_array+0x50/0x88
msi_alloc_desc+0x88/0xd0
...
The buggy address is located 0 bytes to the right of
allocated 48-byte region [ffff8000053a6880, ffff8000053a68b0)
The report is caused by the patch. __msi_domain_alloc_irqs() passes
nvec_aligned (4) to __irq_domain_alloc_irqs() together with
desc->affinity, which has only nvec_used (3) entries. alloc_descs()
then reads all 4 entries. The fourth entry is read past the end of
the array. If the data there has no CPU set, the allocation fails
with -ENOSPC. Without PCI_IRQ_AFFINITY, desc->affinity is NULL, so
this read does not happen.
On the brcmstb path, v2 still frees one vector at a time because
brcmstb on this tree uses pci_msi_create_irq_domain(), and neither
IRQ_DOMAIN_FLAG_MSI_DEVICE nor IRQ_DOMAIN_FLAG_MSI_PARENT is set.
The MIP behind this port has 8 vectors, while the switch ports use 4.
Therefore, the MIP path works for this 4-vector block but not for the
original 5-vector endpoint, which is rounded up to 8.
The test module from my earlier report is below. I bound it using
driver_override and loaded it with min_nvec=1 nvec=3 affinity=1.
// SPDX-License-Identifier: GPL-2.0
// TEST-ONLY Multi-MSI exerciser. Bind to a PCI device via
// driver_override and request a parametrized MSI vector count to probe
// brcmstb/core accounting. nvec: max vectors, min_nvec: floor,
// affinity: add PCI_IRQ_AFFINITY (triggers the managed-affinity path),
// request: request_irq the granted vectors.
#include <linux/module.h>
#include <linux/pci.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
static int nvec = 5;
module_param(nvec, int, 0644);
static int min_nvec = 1;
module_param(min_nvec, int, 0644);
static int affinity;
module_param(affinity, int, 0644);
static int request;
module_param(request, int, 0644);
static irqreturn_t msi_exer_isr(int irq, void *data)
{
return IRQ_HANDLED;
}
static int msi_exer_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
{
unsigned int flags = PCI_IRQ_MSI;
int ret, i, n;
ret = pci_enable_device(pdev);
if (ret)
return ret;
pci_set_master(pdev);
if (affinity)
flags |= PCI_IRQ_AFFINITY;
n = pci_alloc_irq_vectors(pdev, min_nvec, nvec, flags);
if (n < 0) {
dev_info(&pdev->dev,
"msi_exer: alloc(%d..%d,aff=%d) FAILED %d\n",
min_nvec, nvec, affinity, n);
pci_disable_device(pdev);
return n;
}
dev_info(&pdev->dev,
"msi_exer: req %d..%d aff=%d -> got=%d base_irq=%d\n",
min_nvec, nvec, affinity, n, pci_irq_vector(pdev, 0));
for (i = 0; i < n; i++) {
int irq = pci_irq_vector(pdev, i);
struct irq_data *d = irq_get_irq_data(irq);
dev_info(&pdev->dev,
"msi_exer: vec %d irq %d hwirq 0x%lx\n", i, irq,
d ? (unsigned long)irqd_to_hwirq(d) : 0UL);
if (request && request_irq(irq, msi_exer_isr, 0,
"msi_exer", pdev))
dev_info(&pdev->dev,
"msi_exer: request_irq %d failed\n",
irq);
}
pci_set_drvdata(pdev, (void *)(long)n);
return 0;
}
static void msi_exer_remove(struct pci_dev *pdev)
{
int n = (int)(long)pci_get_drvdata(pdev);
int i;
if (request)
for (i = 0; i < n; i++)
free_irq(pci_irq_vector(pdev, i), pdev);
pci_free_irq_vectors(pdev);
pci_disable_device(pdev);
dev_info(&pdev->dev, "msi_exer: removed (freed %d)\n", n);
}
static const struct pci_device_id msi_exer_ids[] = { { 0 } };
MODULE_DEVICE_TABLE(pci, msi_exer_ids);
static struct pci_driver msi_exer_driver = {
.name = "msi_exer",
.id_table = msi_exer_ids,
.probe = msi_exer_probe,
.remove = msi_exer_remove,
};
module_pci_driver(msi_exer_driver);
MODULE_LICENSE("GPL");
Thanks,
Sangwoo
2026년 10월 14일 (수) 오전 1:06, Bjorn Helgaas <helgaas at kernel.org>님이 작성:
>
> On Tue, Oct 13, 2026 at 05:47:49PM +0900, Han / 한상우Sangwoo wrote:
> > Hi Bjorn, Thomas,
> >
> > > > Good. Can you please retest with the updated patch?
> > >
> > > Sangwoo, any update on this?
> >
> > Sorry for the silence.
> >
> > I have not been able to retest yet. I have several work items on a
> > tight schedule right now, and I am currently working on a different
> > board for another project, so I cannot get to this at the moment.
> >
> > For the KASAN slab-out-of-bounds report, I will follow up within
> > October.
> >
> > For the retest with the updated patch, I cannot give a date right now.
> > I am sorry about that.
>
> No worries, just wanted to make sure we didn't drop this. I have a
> note to follow up again later :)
>
> > 2026년 10월 13일 (화) 오전 6:47, Bjorn Helgaas <helgaas at kernel.org>님이 작성:
> > >
> > > On Thu, Oct 01, 2026 at 09:29:27PM +0200, Thomas Gleixner wrote:
> > > > On Mon, Sep 21 2026 at 18:05, Han / 한상우Sangwoo wrote:
> > > > > I tested Thomas's patch on the same hardware setup with the 5-vector
> > > > > MSI endpoint. The patch was applied as posted on top of 6.12.93
> > > > > (rpi-6.12.y).
> > > > >
> > > > > With the patch applied:
> > > > >
> > > > > - The MSI base hwirq remained at 0x8 across 200 driver reload cycles.
> > > > > - The driver got all 5 requested vectors on every cycle, with no
> > > > > single-MSI fallback. Multiple Message Enable remained at 8.
> > > > > - The brcmstb inner-domain mapping returned to baseline after each
> > > > > unload/reload, with 12 mapped while the driver was loaded and 4 after
> > > > > unload.
> > > > > - A kprobe showed one allocation of 8 vectors followed by eight
> > > > > single-vector frees, with nothing left over.
> > > >
> > > > 8 single vector frees?
> > > >
> > > > Seems I got something wrong there verus the bulk free. Updated patch
> > > > below.
> > > >
> > > > > The MSI vector exhaustion issue I originally observed no longer
> > > > > reproduces with the patch.
> > > >
> > > > Good. Can you please retest with the updated patch?
> > >
> > > Sangwoo, any update on this?
> > >
> > > I don't see this patch in linux-next (next-20261012), so I don't think
> > > this will be fixed for v7.4 unless something moves it ahead.
> > >
> > > > > I also observed a KASAN report with managed affinity. Using a small
> > > > > out-of-tree test module bound to the same endpoint and requesting 1..3
> > > > > vectors with PCI_IRQ_MSI | PCI_IRQ_AFFINITY, a request for 3 vectors
> > > > > resulted in:
> > > > >
> > > > > BUG: KASAN: slab-out-of-bounds in __irq_alloc_descs+0x158/0x460
> > > > >
> > > > > Requests for 5 and 7 vectors were capped to 4 on this 4-CPU system and
> > > > > did not trigger the report. I have not checked this against the
> > > > > unpatched kernel yet, so I cannot tell whether it is related to the
> > > > > patch.
> > > >
> > > > Any updates on that? Also please provide the source for that test.
> > > >
> > > > Thanks,
> > > >
> > > > tglx
> > > > ---
> > > > --- a/kernel/irq/irqdomain.c
> > > > +++ b/kernel/irq/irqdomain.c
> > > > @@ -1610,6 +1610,17 @@ static void irq_domain_free_irqs_hierarc
> > > > if (!domain->ops->free)
> > > > return;
> > > >
> > > > + /*
> > > > + * MSI device domains are capable of bulk free.
> > > > + *
> > > > + * CHECKME: Are all MSI parent domains capable?
> > > > + */
> > > > + if (domain->flags & (IRQ_DOMAIN_FLAG_MSI_DEVICE | IRQ_DOMAIN_FLAG_MSI_PARENT)) {
> > > > + if (irq_domain_get_irq_data(domain, irq_base))
> > > > + domain->ops->free(domain, irq_base, nr_irqs);
> > > > + return;
> > > > + }
> > > > +
> > > > for (i = 0; i < nr_irqs; i++) {
> > > > if (irq_domain_get_irq_data(domain, irq_base + i))
> > > > domain->ops->free(domain, irq_base + i, 1);
> > > > --- a/kernel/irq/msi.c
> > > > +++ b/kernel/irq/msi.c
> > > > @@ -1333,20 +1333,28 @@ static int __msi_domain_alloc_irqs(struc
> > > >
> > > > ops->set_desc(&arg, desc);
> > > >
> > > > - virq = __irq_domain_alloc_irqs(domain, -1, desc->nvec_used,
> > > > + /* Make sure a MULTI-MSI allocation is power of two */
> > > > + unsigned int nvec_aligned = roundup_pow_of_two(desc->nvec_used);
> > > > +
> > > > + virq = __irq_domain_alloc_irqs(domain, -1, nvec_aligned,
> > > > dev_to_node(dev), &arg, false,
> > > > desc->affinity);
> > > > if (virq < 0)
> > > > return msi_handle_pci_fail(domain, desc, allocated);
> > > >
> > > > - for (i = 0; i < desc->nvec_used; i++) {
> > > > + for (i = 0; i < nvec_aligned; i++) {
> > > > irq_set_msi_desc_off(virq, i, desc);
> > > > irq_debugfs_copy_devname(virq + i, dev);
> > > > ret = msi_init_virq(domain, virq + i, vflags);
> > > > if (ret)
> > > > return ret;
> > > > }
> > > > +
> > > > if (info->flags & MSI_FLAG_DEV_SYSFS) {
> > > > + /*
> > > > + * This only exposes desc->nvec_used and ignores the
> > > > + * overallocated MULTI-MSI ones.
> > > > + */
> > > > ret = msi_sysfs_populate_desc(dev, desc);
> > > > if (ret)
> > > > return ret;
> > > > @@ -1610,13 +1618,15 @@ static void __msi_domain_free_irqs(struc
> > > > continue;
> > > >
> > > > /* Make sure all interrupts are deactivated */
> > > > - for (i = 0; i < desc->nvec_used; i++) {
> > > > + unsigned int nvec_aligned = roundup_pow_of_two(desc->nvec_used);
> > > > +
> > > > + for (i = 0; i < nvec_aligned; i++) {
> > > > irqd = irq_domain_get_irq_data(domain, desc->irq + i);
> > > > if (irqd && irqd_is_activated(irqd))
> > > > irq_domain_deactivate_irq(irqd);
> > > > }
> > > >
> > > > - irq_domain_free_irqs(desc->irq, desc->nvec_used);
> > > > + irq_domain_free_irqs(desc->irq, nvec_aligned);
> > > > if (info->flags & MSI_FLAG_DEV_SYSFS)
> > > > msi_sysfs_remove_desc(dev, desc);
> > > > desc->irq = 0;
More information about the linux-arm-kernel
mailing list