Question about potential missed IPI events

[Bo Gan]

On 11/10/23 10:05 PM, Bo Gan wrote:
> Hi OpenSBI community,
> 
> Current IPI send/recv implementation uses ipi_data->ipi_type to indicate
> an IPI event pending. In sbi_ipi_send, I see have the following:
> 
>> /*
>>  * Set IPI type on remote hart's scratch area and
>>  * trigger the interrupt
>>  */
>> atomic_raw_set_bit(event, &ipi_data->ipi_type);
>> smp_wmb();
>>
>> if (ipi_dev && ipi_dev->ipi_send)
>>     ipi_dev->ipi_send(remote_hartindex)
> 
> And in sbi_ipi_process, the following:
> 
>> if (ipi_dev && ipi_dev->ipi_clear)
>>     ipi_dev->ipi_clear(hartindex);
>>
>> ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0);
>> ipi_event = 0;
>> while (ipi_type) {
> 
> I'm think of if there's any possibility of missed IPIs. Assuming we are
> talking about sifive/clint (aclint_mswi.c), I know it means that there're
> both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit
> and the write to remote hart's msip register in clint. Consider the following
> 
> Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B,
> Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit,
> resulting in both IPI (from B and C) coalesced to a single one. In this case,
> A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI
> was indeed generated by B's write to msip. However will the barrier ensure A
> also observe the write to ipi_type from hart C? For the coalescing to happen,
> clint will observe C's write-1 before A's write-0, does it imply A'll observe
> C's write to ipi_type from xchg?
> 
> I doubt it mainly due to 2 reasons:
> 
> 1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't
>     think it helps preventing the writel to happen before xchg. If that happens,
>     it's possible that A missed C's write to ipi_type.
> 
> 2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a
>     write, not a read. A wasn't reading from msip, but only writing to it. I'm
>     not sure if A's write to msip after C's write (observed by clint) implies
>     A's observing the same order. A is not reading from msip, so what helps to
>     establish this order? Will the RVWMO global memory order help? Can we think
>     of clint as another hart that follows RVWMO?
> 
> Bo
> 

Hi @Atish @Anup and other folks, any thoughts on this?