[PATCH v2 0/3] PCI: endpoint: Support hardware-owned MSI-X table and PBA

[Koichiro Den]

On Mon, Aug 31, 2026 at 10:01:31PM +0200, Niklas Cassel wrote:
> Hello Koichiro,
> 
> On Tue, Sep 01, 2026 at 01:11:09AM +0900, Koichiro Den wrote:
> > This is not driven by a performance target, so I do not have numbers. The
> > immediate issue is correctness. ntb_hw_epf can select MSI-X, but pci-epf-vntb
> > currently configures and raises MSI. I want that MSI-X path to work on RK3588
> > with the controller doorbell. I will make it clear in the next cover letter
> > that, with this series, a doorbell sent from ntb_tool on the pci-epf-vntb side
> > of RK3588 reaches ntb_tool on the ntb_hw_epf host side when MSI-X is selected.
> > I believe that is the simplest E2E test showing what this series fixes.
> 
> Okay, if the main issue is that pci-epf-vntb does not raise MSI-X, even when
> ntb_hw_epf host side has requested MSI-X, then I think that the first patch
> in the series should fix this, so that the fix can easily be backported.
> 
> (Right now you fix this in patch 3/3, and the fix in patch 3/3 depends on
> both patches 1/3 and 2/3.)
> 
> I think that pci-epf-vntb actually respecting the configured IRQ type by the
> host is a separate logical change from adding support for HW defined layout.

Sure. It can be a separate patch, first making pci-epf-vntb use an EPF-owned
MSI-X Table/PBA like the other in-tree EPF drivers with MSI-X support.

> 
> 
> > I initially considered making the hardware-owned layout automatic for every EPF.
> > pci-epf-ntb (not vNTB!) is the reason I did not. It currently reads
> > host-programmed MSI-X entries through epf_bar[] backing and uses them to set up
> > peer outbound mappings. A hardware-owned Table has no such backing, and the
> > current EPC API has no generic way to read those entries. I suspect that
> > supporting it there would require a broader design change. Also, I do not have
> > hardware for the pci-epf-ntb bridge configuration.
> > 
> > pci-epf-test and nvmet-pci-epf should be easier to convert. However, converting
> > only those would not remove the per-layout handling while pci-epf-ntb still uses
> > an EPF-owned Table. I would prefer to handle those separately when they can be
> > properly tested. This is what I meant by this part of the cover letter:
> > 
> >      [...]
> > 
> >      layouts. This avoids unnecessary changes and reduces regression risk.
> >      They can use a hardware-owned layout later if/when needed.
> 
> Okay, if pci-epf-ntb is special, perhaps we should add a comment in
> pci-epf-ntb which explains why it cannot support HW defined layout.
> 
> But for consistency, if we modify pci-epf-vntb, I think it would make sense
> to also modify pci-epf-test and nvmet-pci-epf as well in the same series.
> 
> pci-epf-test should be easy to test, just run the pci endpoint selftest.
> 
> nvmet-pci-epf should be quite easy to test too, just run the normal fio based
> testing that we usually run, and unless you see worse performance, I think we
> are good. (If we see worse performance, that could be an indication that one
> of the MSI-X for the different completion queues are not being correctly
> triggered. By default, the host side nvme driver creates one completion queue
> (and one submission queue) per host CPU.

Fair enough. As an experiment, I converted nvmet-pci-epf and tries to test it. I
could not get as far as running fio because host-side NVMe initialization timed
out.

I found that on RK3588, host writes through BAR4 to the Message Address field
(the first 8 bytes) of entry #0 do not seem to show up in the internal iMSIX-TX
Table RAM. The rest of entry #0 and the entries from entry #1 onward that I
tested seem fine. The same two DWORDs can be written through direct DBI Table
RAM access, and the doorbell for entry #0 then works.

I'm not sure but one thing I wondered is whether this could be related to the
boundary between the BAR4-mapped unrolled iATU registers and the MSI-X Table. On
RK3588, the iATU register window starts at 0x2000 and the MSI-X Table
immediately follows at 0x4000. If there somehow is an address decoding issue at
that boundary, with the first part of the Table being taken as an iATU access, I
would like to know.

This still sounds really odd, and it does not obviously explain why both DWORDs
at 0x4000 and 0x4004 are affected. I may be missing something important. If
anything similar involving iMSIX-TX Table RAM entry #0 has been discussed
before, please point me to the relevant thread.

(Also, I can share my current WIP v3 series, which makes pci-epf-test and
nvmet-pci-epf use the hardware-owned layout when available, while leaving
pci-epf-ntb with its EPF-owned layout. If you have any setup where you could
test it, please let me know.)

> 
> 
> What I suggest that you instead do something like:
> Patch 1/7: pci-epf-vntb: Fix to actually use MSI-X when requested.
>            This probably includes adding the code that is inside the
> 	   if (ret == -ENOENT) { }.
> Patch 2/7: PCI: endpoint: Support hardware-owned MSI-X table and PBA
> Patch 3/7: PCI: dw-rockchip: Support fixed MSI-X table and PBA on RK3588
> Patch 4/7: pci-epf-vntb: Add code that calls pci_epc_get_hw_msix_layout()
>            and uses that layout, if the function returned success.
> Patch 5/7: pci-epf-test: Add code that calls pci_epc_get_hw_msix_layout()
>            and uses that layout, if the function returned success.
> Patch 6/7: nvmet-pci-epf: Add code that calls pci_epc_get_hw_msix_layout()
>            and uses that layout, if the function returned success.
> Patch 7/7: pci-epf-ntb: Add comment that explains why it cannot call
>            pci_epc_get_hw_msix_layout() like all other EPF drivers that
> 	   support MSI-X.
> 
> 
> This way, patch 1/7 can be backported, and will help all users vNTB users,
> even the users that do not use RK3588 as an EPC.

That split makes sense. However, given the issue I described above, perhaps I
should send patch 1/7 as a standalone fix and hold off on the rest until the
issue is understood and resolved.

Testing vNTB alone (ie. this v2 way) did not expose it because its first data
doorbell uses MSI-X Table entry #2. I only found the issue because your
suggestion led me to convert and test nvmet-pci-epf, thank you!

> 
> 
> > > I understand that you introduce dw_pcie_ep_msix_layout_is_hw_owned()
> > > because you want an EPF driver optionally use the HW defined table.
> > > But if all EPF drivers always use the HW defined table if available, I think
> > > you can avoid introducing this helper, and let rockchip_pcie_raise_irq()
> > > unconditionally call dw_pcie_ep_raise_msix_irq_doorbell() for case PCI_IRQ_MSIX.
> > 
> > Yes, under that assumption I agree. With the current mix, however, the Rockchip
> > callback still has to choose the helper according to the selected layout. That
> > is why the DWC code records which layout was selected.
> > 
> > I will update the cover letter to state the motivation and this boundary more
> > directly. Does keeping the layout choice with each EPF for this reason sound
> > reasonable?
> 
> Yes, since pci-epf-ntb apparently is special, I can see why you need
> both ep_func->msix_hw_owned and dw_pcie_ep_msix_layout_is_hw_owned().

Sounds good.
Thanks for taking a look!

Best regards,
Koichiro

> 
> 
> Kind regards,
> Niklas