Question about potential missed IPI events

[Xiang W]

在 2023-11-15星期三的 16:33 -0800，Bo Gan写道：
> On 11/10/23 10:05 PM, Bo Gan wrote:
> > Hi OpenSBI community,
> > 
> > Current IPI send/recv implementation uses ipi_data->ipi_type to indicate
> > an IPI event pending. In sbi_ipi_send, I see have the following:
> > 
> > > /*
> > >  * Set IPI type on remote hart's scratch area and
> > >  * trigger the interrupt
> > >  */
> > > atomic_raw_set_bit(event, &ipi_data->ipi_type);
> > > smp_wmb();
> > > 
> > > if (ipi_dev && ipi_dev->ipi_send)
> > >     ipi_dev->ipi_send(remote_hartindex)
> > 
> > And in sbi_ipi_process, the following:
> > 
> > > if (ipi_dev && ipi_dev->ipi_clear)
> > >     ipi_dev->ipi_clear(hartindex);
> > > 
> > > ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0);
> > > ipi_event = 0;
> > > while (ipi_type) {
> > 
> > I'm think of if there's any possibility of missed IPIs. Assuming we are
> > talking about sifive/clint (aclint_mswi.c), I know it means that there're
> > both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit
> > and the write to remote hart's msip register in clint. Consider the following
> > 
> > Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B,
> > Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit,
> > resulting in both IPI (from B and C) coalesced to a single one. In this case,
> > A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI
> > was indeed generated by B's write to msip. However will the barrier ensure A
> > also observe the write to ipi_type from hart C? For the coalescing to happen,
> > clint will observe C's write-1 before A's write-0, does it imply A'll observe
> > C's write to ipi_type from xchg?
Does the picture below represent what you mean?

 B             A             C
 |             |             |
 | xchg        |             |
 | send------->|             |   
 |             | xchg        |
 |             |             | xchg
 |             |<------------| send
 |             | clear       |
> 

In fact, A will not lose C's ipi message and will be deleted when it receives
a new ipi interrupt, as shown below
 B             A             C             D
 |             |             |             |
 | xchg        |             |             |
 | send------->|             |             |
 |             | xchg        |             |
 |             |             | xchg        |
 |             |<------------| send        |
 |             | clear       |             |
               |                           |
               |<--------------------------| send

There is another situation that will cause the efficiency to drop. Please refer to
the figure below.
 B             A             C
 |             |             |
 | xchg        |             |
 | send------->|             | xchg  
 |             | clear       |
 |             | xchg        |
 |             |<------------| send
 |             |             |

A may enter exception handling one more time.


I think we need to protect xchg/send and clear/xchg with spinlock.


Regards,
Xiang W
> > I doubt it mainly due to 2 reasons:
> > 
> > 1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't
> >     think it helps preventing the writel to happen before xchg. If that happens,
> >     it's possible that A missed C's write to ipi_type.
> > 
> > 2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a
> >     write, not a read. A wasn't reading from msip, but only writing to it. I'm
> >     not sure if A's write to msip after C's write (observed by clint) implies
> >     A's observing the same order. A is not reading from msip, so what helps to
> >     establish this order? Will the RVWMO global memory order help? Can we think
> >     of clint as another hart that follows RVWMO?
> > 
> > Bo
> > 
> 
> Hi @Atish @Anup and other folks, any thoughts on this?
>