[PATCH] virtio: Try to untangle DMA coherency

[Michael S. Tsirkin]

On Wed, Feb 01, 2017 at 12:25:57PM +0000, Robin Murphy wrote:
> By forcing on DMA API usage for ARM systems, we have inadvertently
> kicked open a hornets' nest in terms of cache-coherency. Namely that
> unless the virtio device is explicitly described as capable of coherent
> DMA by firmware, the DMA APIs on ARM and other DT-based platforms will
> assume it is non-coherent. This turns out to cause a big problem for the
> likes of QEMU and kvmtool, which generate virtio-mmio devices in their
> guest DTs but neglect to add the often-overlooked "dma-coherent"
> property; as a result, we end up with the guest making non-cacheable
> accesses to the vring, the host doing so cacheably, both talking past
> each other and things going horribly wrong.
> 
> To prevent regressing those existing use cases relying on implicit
> coherency, but still fixing the original problem of (coherent PCI)
> legacy devices behind IOMMUs, take the more conservative approach of
> only hitting the DMA API switch for coherent devices, where we can be
> sure it is safe, and preserving the old non-DMA behaviour otherwise.
> This does not affect devices setting the VIRTIO_F_IOMMU_PLATFORM flag,
> which as before are still at the mercy of architecture code correctly
> knowing their coherency, so explicitly call this out in the virtio-mmio
> DT binding in the hope of heading off any further workarounds for future
> firmware mishaps.
> 
> Fixes: c7070619f340 ("vring: Force use of DMA API for ARM-based systems with legacy devices")
> Signed-off-by: Robin Murphy <robin.murphy at arm.com>
> ---
>  Documentation/devicetree/bindings/virtio/mmio.txt |  3 +++
>  drivers/virtio/virtio_ring.c                      | 11 ++++++++---
>  2 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/virtio/mmio.txt b/Documentation/devicetree/bindings/virtio/mmio.txt
> index 5069c1b8e193..8f2a981e1010 100644
> --- a/Documentation/devicetree/bindings/virtio/mmio.txt
> +++ b/Documentation/devicetree/bindings/virtio/mmio.txt
> @@ -7,6 +7,8 @@ Required properties:
>  - compatible:	"virtio,mmio" compatibility string
>  - reg:		control registers base address and size including configuration space
>  - interrupts:	interrupt generated by the device
> +- dma-coherent:	required if the device (or host emulation) accesses memory
> +		cache-coherently, absent otherwise
>  
>  Example:
>  
> @@ -14,4 +16,5 @@ Example:
>  		compatible = "virtio,mmio";
>  		reg = <0x3000 0x100>;
>  		interrupts = <41>;
> +		dma-coherent;
>  	}
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index 7e38ed79c3fc..961af25b385c 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -20,6 +20,7 @@
>  #include <linux/virtio_ring.h>
>  #include <linux/virtio_config.h>
>  #include <linux/device.h>
> +#include <linux/property.h>
>  #include <linux/slab.h>
>  #include <linux/module.h>
>  #include <linux/hrtimer.h>
> @@ -160,10 +161,14 @@ static bool vring_use_dma_api(struct virtio_device *vdev)
>  		return true;
>  
>  	/*
> -	 * On ARM-based machines, the DMA ops will do the right thing,
> -	 * so always use them with legacy devices.
> +	 * On ARM-based machines, the coherent DMA ops will do the right
> +	 * thing, so always use them with legacy devices. However, using
> +	 * non-coherent DMA when the host *is* actually coherent, but has
> +	 * forgotten to tell us, is going to break badly; since this situation
> +	 * already exists in the wild, maintain the old behaviour there.
>  	 */
> -	if (IS_ENABLED(CONFIG_ARM) || IS_ENABLED(CONFIG_ARM64))
> +	if ((IS_ENABLED(CONFIG_ARM) || IS_ENABLED(CONFIG_ARM64)) &&
> +	    device_get_dma_attr(&vdev->dev) == DEV_DMA_COHERENT)
>  		return !virtio_has_feature(vdev, VIRTIO_F_VERSION_1);
>  
>  	return false;

This is exactly what I feared.

Could we identify fastboot and do the special dance just for it?

I'd like to do that instead. It's fastboot doing the unreasonable thing
here and deviating from what every other legacy device without exception
did for years. If this means fastboot will need to update to virtio 1,
all the better.

> -- 
> 2.11.0.dirty