[PATCH 1/4] um: irqs: process outstanding IRQs when unblocking signals

[Johannes Berg]

On Fri, 2023-10-20 at 14:58 +0200, Benjamin Beichler wrote:
> > On Fri, 2023-10-20 at 14:06 +0200, Benjamin Beichler wrote:
> > > Am 20.10.2023 um 13:39 schrieb Johannes Berg:
> > > > On Fri, 2023-10-20 at 12:38 +0200, Benjamin Beichler wrote:
> > > > > Can you explain, why a time travel handler for stdin may be bad? It
> > > > > sounds like you want to avoid it, but I see no immediate problem.
> > > > > 
> > > > I need to read the thread, but this one's easy ;-)
> > > > 
> > > > The thing is that on such a channel you don't send an ACK when you've
> > > > seen that there's something to handle. As a result, the sender will
> > > > continue running while you're trying to request a new schedule entry
> > > > from the controller. As a result, it may run past your new schedule
> > > > entry because it didn't know about it yet (this would likely bring down
> > > > the controller and crash the simulation), or the relative order of the
> > > > two entries is undefined, in the sense that it depends on the process
> > > > scheduling of the host.
> > > Sorry, but I did not get this. What may run past the schedule entry? Is
> > > your assumption, that the "thing" connected to stdin is always totally
> > > unaware of the time travel mode?
> > No, I'm not assuming that, if that's done all bets are off anyway. I'm
> > assuming that both are connected to the controller.
> > 
> > > When our (to be published) simulation send something on serial lines (I
> > > think it does not matter whether it is a socket or a pipe), we expect
> > > that the uml instance needs to be run as long as it changes back to
> > > idle/wait state before the simulation time is advanced. Since the basis
> > > model of the time travel mode is, that you have infinite amount of
> > > processing power, the interrupt needs to be always handled at the
> > > current time.
> > Yes but you need to schedule for the interrupt, and you don't
> > necessarily know what 'current time' is at interrupt time.
> > 
> > So let's say you have "something" that's scheduled to run at times
> >   - 1000
> >   - 2000
> >   - 3000
> > 
> > and free-until is 10000 or something high.
> > 
> > Now it sends a message to Linux stdio at 1000.
> > 
> > But it doesn't have a way to wait for ack. So it continues, checks the
> > schedule and free-until, and can advance time to 2000 since it doesn't
> > yet know Linux requested time to run at 1000.
> 
> Do I get it right, that you anticipate that all parts of the simulation 
> may run at different paces and the controller is only needed for 
> synchronization?

Not sure what you mean by that.

What I _wanted_ is that only one can run at a time (from a host
perspective), so e.g. virtio stuff works like

A is running
A -> B virtio:
A puts data on ring virtio ring
A sends message on vhost-user socket to indicate ring update to B
A waits for ACK for that message
B sees the ring update message
B requests time from controller
B sends ACK to A
A continues running
A gets to the next scheduling point but now sees free-until==now
A releases time slice to controller
controller assigns time slice to B
B runs to process the virtio ring data


So you see that only one is running at a time.

Without the ACK, you'd have
 B sees the ring update message
 B requests time from the controller

in parallel with

 A continues running
 A gets to the next scheduling point

but it might either not see free-until==now, or other things.


> My mind model is, that simulation time advances in the controller are 
> only done, If all parts of the simulation reached the current simulation 
> time and are at the event processed/idle/wait state. 

Yeah that's a simpler model, and in what we implemented it's true if you
do have the ACK messages. If don't have free-until, that's also possibly
true, although we might have implementation bugs in a sense with
time_travel_add_irq_event() since that looks at the current time. And
also, if you request something the controller thinks is already in the
past because the other side updated the controller due to free-until,
the simulation just crashes.

It's also possible that B (in the situation above) actually *does* ask
the controller who should run next, and the controller simply doesn't
yet know:

 B sends message to A's stdin (say at 1000)
 B requests to run at 2000 from controller
 B releases time to controller
 controller sets time to 2000 and tells B to run
 A requests scheduling at 1000
 controller *crashes*

This is actually quite likely if the host is running 6.6 with the new
EEVDF scheduler, because it will prefer to let B continue running, and
then it just needs to pick the controller over A next.

This again is because of the lack of synchronization in editing the
schedule (sending messages to the controller) which the ACK message
solves.

Hence our internal use cases for "text protocol" actually use virtio-
console rather than stdio, so you do have ACK on the vhost-user protocol
again.

> Therefore, the 
> thing connected to stdio will not advance to some other time, if some 
> other component has outstanding events (i.e., an interrupt at this 
> case). 

Yes, but by the time it actually asks "are there outstanding events",
they might no be added yet, due to hos scheduling order!

> Of course free-running is a problematic in this mind model, but 
> in this mode you willingly sacrifice the precision of events for 
> performance.

If you're referring to free-until: that isn't free-running, it's just
meant to reduce the communication overhead if you know nobody else is
running. The controller maintains this.

Btw we have further overhead reductions with shared memory, need to push
all that out.

> > > Maybe we need to define, a bit more formally, how the (designed)
> > > processing model of interrupts in time travel mode is.
> > That's probably a separate Master's thesis ;-)
> 
> It could, but some basic definition for a common understanding might help.

:)
Does the above help a bit? But yeah maybe I should try to write it down
more formally.

> However, do you like to have a look at my current state of time travel 
> changes in github or do you think an RFC to the mailing list is better? 
> I like github for a quick dive into patches, but your whole workflow is 
> maybe different ;-)
> 

RFC to the list is probably easier to comment on etc.

johannes