Question about potential missed IPI events

[Bo Gan]

On 11/20/23 8:34 PM, Anup Patel wrote:
>>
>> I doubt it mainly due to 2 reasons:
>>
>> 1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't
>>      think it helps preventing the writel to happen before xchg. If that happens,
>>      it's possible that A missed C's write to ipi_type.
>>
>> 2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a
>>      write, not a read. A wasn't reading from msip, but only writing to it. I'm
>>      not sure if A's write to msip after C's write (observed by clint) implies
>>      A's observing the same order. A is not reading from msip, so what helps to
>>      establish this order? Will the RVWMO global memory order help? Can we think
>>      of clint as another hart that follows RVWMO?
> 
> The clint HW is supposed to handle writes coming from multiple
> harts so we can safely assume that multiple writes reaching to
> clint will be completed in the order in which clint observes them.
> 
> Things can only break if hart A does xchg() out-of-order before
> writel() issued by ipi_clear() completes.
> 
> This can be understood by an example which is as follows:
> 1) hart A orders xchg() before writel() issued by ipi_clear()
> 2) hart B is about to do set_bit() for hart A ipi_type but
>      hart A does xchg() before hart B does set_bit() so
>      hart A will not observe anything in ipi_type
> 3) hart B does set_bit() and ipi_send() before hart A does
>      writel() from ipi_clear()
> 
> The above result is hart A totally missing IPI from hart B but
> this is only possible if hart A orders xchg() before writel() from
> ipi_clear(). This is a very very rare scenario not observed on
> any RISC-V implementations so far but to prevent this we
> need a matching smp_wmb() between ipi_clear() and xchg()
> in sbi_ipi_process().
> 

Hi Anup,

Thank you so much for the detailed explanation. I totally get your point.
I just have one doubt about smp_wmb. In OpenSBI, smp_wmb is RISCV_FENCE(w,w).
If I understand correctly, it won't enforce MMIO order. Given your analysis,
I assume you meant wmb(), which is RISCV_FENCE(ow,ow). Am I right? Or do we
want more precise fences, such as:

  RISCV_FENCE(w,o) between set_bit(ipi_type) and ipi_dev->ipi_send in Hart B
  RISCV_FENCE(o,w) between ipi_dev->ipi_clear and xchg(ipi_type) in Hart A

Let me know your thoughts, and I can make a patch and send to mailing list.

Bo