[RFC] virtio: use mandatory barriers for remote processor vdevs

[Ohad Ben-Cohen]

On Tue, Nov 29, 2011 at 3:11 PM, Michael S. Tsirkin <mst at redhat.com> wrote:
> On Tue, Nov 29, 2011 at 02:31:26PM +0200, Ohad Ben-Cohen wrote:
>> Virtio is using memory barriers to control the ordering of
>> references to the vrings on SMP systems. When the guest is compiled
>> with SMP support, virtio is only using SMP barriers in order to
>> avoid incurring the overhead involved with mandatory barriers.
>>
>> Lately, though, virtio is being increasingly used with inter-processor
>> communication scenarios too, which involve running two (separate)
>> instances of operating systems on two (separate) processors, each of
>> which might either be UP or SMP.
>
> Is that using virtio-mmio?

No, I'm using this:

https://lkml.org/lkml/2011/10/25/139

> Sorry, could you pls explain what are 'two external processors'?
> I think I know that if two x86 CPUs in an SMP system run kernels built
> in an SMP configuration, smp_*mb barriers are enough.

Sure:

My setup is not SMP-based; it's two separate processors running in AMP
configuration. The processors have completely different architectures,
are not cache coherent, and only simply share some memory, which is
used for communications using virtio as the shared memory "wire"
protocol (i.e. we're not even doing virtualization: we have Linux on
one processor, and some RTOS on another processor, and they use virtio
to send and receive buffers).

So it's not SMP effects we're controlling; we're pretty much doing
MMIO and must use mandatory barriers (otherwise we see breakage).

> Is an extra branch faster or slower than reverting d57ed95?

Sorry, unfortunately I have no way to measure this, as I don't have
any virtualization/x86 setup. I'm developing on ARM SoCs, where
virtualization hardware is coming, but not here yet.

> One wonders how the remote side knows enough to set this flag?

The remote side is a dedicated firmware loaded in order to boot the
other processor, so it really is intimate with the platform and its
requirements (and it publishes the virtio device features for every
supported virtio device).

>> Though I also wonder how big really is the perforamnce gain of d57ed95 ?
>
> Want to check and tell us?

Sorry, as I said, I have no way to do that... Any chance you did some
measurements back then when d57ed95 was introduced ?

I just guessed it did improve performance, otherwise you probably
wouldn't have bothered. But if not, then reverting it is surely
preferable to adding more complexity.

Thanks!
Ohad.