REGRESSION in 6.0-rc7 caused by patch "ARM/dma-mapping: use dma-direct unconditionally"

[Marek Behún]

On Tue, 4 Oct 2022 09:10:00 +0200
Christoph Hellwig <hch at lst.de> wrote:

> On Mon, Oct 03, 2022 at 11:30:31PM +0200, Marcin Wojtas wrote:
> > I have one overall concern here. On all kinds of A38x-based boards I
> > worked on, by default, the firmware set all devices (e.g. network,
> > AHCI, XHCI) on MBUS as fully IO cache coherent - it should be
> > reflected in the MVNETA_WIN_BASE(w) registers attribute field. Bits
> > [15:8] should be set to 0x1D (or 0x1E if there is a second DRAM CS
> > used). Can you please try adding 'dma-coherent;' property under the
> > 'internal-regs' node?  
> 
> Robin mentioned something similar earlier.  This almost smalls like
> we somehow manage to mark these device non-coherent by accident now.
> 
> The interesting part of the bisected commit is the change to
> mvebu_hwcc_notifier that used to force the DMA OPS to
> arm_coherent_dma_ops, but now just sets the ->dma_coherent flags,
> which seems to get overriden somehow again.  Maybe the notifier is
> run before arch_setup_dma_ops, even if that seems odd?  As that is
> the only thing I could think of, maybe try this patch:
> 
> 
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index 089c9c644cce2..76789650e2596 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -1770,7 +1770,9 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>  			const struct iommu_ops *iommu, bool coherent)
>  {
>  	dev->archdata.dma_coherent = coherent;
> -	dev->dma_coherent = coherent;
> +
> +	if (coherent)
> +		dev->dma_coherent = true;
>  
>  	/*
>  	 * Don't override the dma_ops if they have already been set. Ideally
> 

Indeed this fixes the issue, and indeed arch_setup_dma_ops is called
later than mvebu_hwcc_notifier.

  bash-5.1# dmesg | grep -Ee '(mvebu_hwcc|arch_setup)' | grep ethernet
  [    0.009350] mvebu-coherency:         mvebu_hwcc_notifier f1070000.ethernet
  [    0.009434] mvebu-coherency:         mvebu_hwcc_notifier f1030000.ethernet
  [    0.009523] mvebu-coherency:         mvebu_hwcc_notifier f1034000.ethernet
  [    1.859657]  arch_setup_dma_ops f1070000.ethernet coherent=0
  [    1.874852]  arch_setup_dma_ops f1030000.ethernet coherent=0
  [    1.889770]  arch_setup_dma_ops f1034000.ethernet coherent=0

But mvebu_hwcc_notifier is called even for device not in internal-regs:\

  bash-5.1# dmesg | grep -Ee '(mvebu_hwcc|arch_setup)'
  [    0.006475] mvebu-coherency:         mvebu_hwcc_notifier pmu
  [    0.006536] mvebu-coherency:         mvebu_hwcc_notifier soc
  [    0.007902] mvebu-coherency:         mvebu_hwcc_notifier fff00000.bootrom
  [    0.007946] mvebu-coherency:         mvebu_hwcc_notifier soc:internal-regs
  ...
  [    0.010790] mvebu-coherency:         mvebu_hwcc_notifier soc:pcie

This probably means that to fix it, we just need to
  select OF_DMA_DEFAULT_COHERENT
in
  config MACH_MVEBU_V7
?

That way it will work with existing device-trees.

Marek