[Question] Possible tlb_sync hang when remote SFENCE.VMA races with HSM hart_stop

[Alex Mao]

Hi Anup, Bin,

On Fri, Jul 3, 2026 at 5:03 AM Anup Patel <anup at brainfault.org> wrote:
> My suggestion is to call sbi_ipi_process() just before
> sbi_platform_early_exit() in sbi_exit(). This way the HART which is
> being stopped will process the pending IPIs before going down.

Thanks, I tried that on current master (3593a5f) with QEMU virt
(-M virt -smp 4 -m 256M, qemu-system-riscv64 9.1.1).

A small S-mode payload starts a neighbor hart, asks it to HART_STOP,
and immediately issues sbi_remote_sfence_vma to that hart. There is
no extra delay in sbi_ipi_send_many(). Local debug prints were added
only to show the enqueue and the tlb_sync stall; they are not in
upstream.

On unmodified master, iter 0 returns and the victim is already
STOPPED. iter 1 hangs:

  REPRO: HSM-stop x RFENCE tlb_sync race boot=0 victim=1
  iter 0
  RFENCE returned err=0
  victim status=1
  iter 1
  tlb_update: enqueue to hart1 in state 1
  HANG: tlb_sync stuck on hart0 count=1

state 1 is SBI_HSM_STATE_STOPPED. The sender still snapshots the
victim as interruptible, then enqueues and increments tlb_sync after
the victim has finished sbi_exit().

With only the suggested sbi_ipi_process() before
sbi_platform_early_exit(), the same command and payload still hang
at iter 1 with the same enqueue-to-STOPPED line.

The extra drain helps when an IPI is already pending. In this run
the enqueue happens after the last drain. sbi_ipi_exit() already
calls sbi_ipi_process() later in sbi_exit(), so moving that call
earlier does not close this window.

Skipping a target that can no longer process IPIs looks acceptable
for RFENCE: a stopped hart will not return to the old S-mode page
tables, and the next HART_START is a new context. I can send an RFC
that serializes the interruptible check with enqueue on the stop
path, if that direction sounds right.

Happy to adjust the approach if you would rather keep the send path
unchanged and only tighten the stop/drain side.

Thanks,
Mao Weiming