[RFC] Supervisor IMSIC state handoff across OpenSBI domains needs MSI routing coordination

[whensun]

From: Theodorus Wensan Februanto <theodoruswensanfebruanto025 at gmail.com>

I am following up on the missing CSR preservation work in OpenSBI issue #433.
The previously submitted AIA interrupt priority and CTR RFC patches pass targeted
A -> B -> A domain-switch tests.

I also prototyped saving and restoring the supervisor IMSIC eidelivery,
eithreshold, EIP, and EIE state in sbi_domain_context.c. The prototype passes
basic CSR and MMIO MSI preservation tests, but a deterministic test exposed a
correctness issue in the EIP restoration path:

  1. Domain A has EIP0 = 0x4 in its saved state.
  2. A test-only MSI with interrupt ID 3 arrives while restoring A.
  3. EIP0 becomes 0x8 before restoration of A's pending snapshot.
  4. Writing EIP0 = 0x4 restores A's bit 2 but drops the new bit 3.

The runtime trace was:

  IMSIC_RACE|AFTER_MSI|0x8
  IMSIC_RACE|AFTER_RESTORE|0x4

The associated regression test expected both bits (0xc), but observed 0x4.
This injection proves a destructive pending-state overwrite in the prototype;
it is not a full test of concurrent device/IOMMU timing.

A simple CSRRS/OR in place of CSRW would avoid this particular lost-pending-bit
interleaving, but it cannot establish whether the new interrupt belongs to
Domain A or Domain B. The supervisor IMSIC interrupt file is shared by the
S-mode domains on the same hart. EIDELIVERY=0 disables delivery, not MSI
recording. Correct state handoff therefore appears to require software
ownership of interrupt identities and coordination with the APLIC/IOMMU/device
MSI routing, including synchronization of straggler writes.

A safe interim behavior is to reject an attempt to switch domains sharing
an IMSIC supervisor file unless such routing coordination is available.
I have verified the guard can be added before any context-switch state is
modified. This intentionally disables that configuration; it does not
constitute a working IMSIC virtualization implementation.

Could you advise whether OpenSBI domain context switching should manage
supervisor IMSIC files at all, or whether ownership/routing and synchronization
should belong to a different layer? If OpenSBI should manage it, is there an
existing platform/irqchip interface for quiescing and rerouting incoming MSIs?

I am withholding the destructive-save/restore prototype from merge review
until there is agreement on the ownership and handoff mechanism.

Thanks,
Theodorus Wensan Februanto