Question about potential missed IPI events

[Bo Gan]

Hi OpenSBI community,

Current IPI send/recv implementation uses ipi_data->ipi_type to indicate
an IPI event pending. In sbi_ipi_send, I see have the following:

> /*
>  * Set IPI type on remote hart's scratch area and
>  * trigger the interrupt
>  */
> atomic_raw_set_bit(event, &ipi_data->ipi_type);
> smp_wmb();
> 
> if (ipi_dev && ipi_dev->ipi_send)
> 	ipi_dev->ipi_send(remote_hartindex)

And in sbi_ipi_process, the following:

> if (ipi_dev && ipi_dev->ipi_clear)
> 	ipi_dev->ipi_clear(hartindex);
> 
> ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0);
> ipi_event = 0;
> while (ipi_type) {

I'm think of if there's any possibility of missed IPIs. Assuming we are
talking about sifive/clint (aclint_mswi.c), I know it means that there're
both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit
and the write to remote hart's msip register in clint. Consider the following

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B,
Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit,
resulting in both IPI (from B and C) coalesced to a single one. In this case,
A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI
was indeed generated by B's write to msip. However will the barrier ensure A
also observe the write to ipi_type from hart C? For the coalescing to happen,
clint will observe C's write-1 before A's write-0, does it imply A'll observe
C's write to ipi_type from xchg?

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't
    think it helps preventing the writel to happen before xchg. If that happens,
    it's possible that A missed C's write to ipi_type.

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a
    write, not a read. A wasn't reading from msip, but only writing to it. I'm
    not sure if A's write to msip after C's write (observed by clint) implies
    A's observing the same order. A is not reading from msip, so what helps to
    establish this order? Will the RVWMO global memory order help? Can we think
    of clint as another hart that follows RVWMO?

Bo