REGRESSION in 6.0-rc7 caused by patch "ARM/dma-mapping: use dma-direct unconditionally"

[Marek Behún]

On Fri, 30 Sep 2022 17:41:44 +0100
Robin Murphy <robin.murphy at arm.com> wrote:

> On 2022-09-30 16:02, Marek Behún wrote:
> > On Fri, 30 Sep 2022 16:52:34 +0200
> > Marek Behún <kabel at kernel.org> wrote:
> >   
> >> On Fri, 30 Sep 2022 14:46:06 +0100
> >> Robin Murphy <robin.murphy at arm.com> wrote:
> >>  
> >>> On 2022-09-30 14:10, Marek Behún wrote:  
> >>>> Hello Linus, Arnd, Robin and Christoph,
> >>>>
> >>>> I just bisected a regression on Turris Omnia (Armada 385), wherein the
> >>>> system hangs shortly after init is run, to commit
> >>>>
> >>>>     ae626eb97376 ("ARM/dma-mapping: use dma-direct unconditionally")
> >>>>     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ae626eb97376
> >>>>
> >>>> In order to fix the regression, I had to revert this commit and
> >>>> subsequent 3 commits:
> >>>>     ae626eb97376 ("ARM/dma-mapping: use dma-direct unconditionally")
> >>>>     42998ef08aba ("ARM/dma-mapping: drop .dma_supported for IOMMU ops")
> >>>>     d563bccfa35b ("ARM/dma-mapping: consolidate IOMMU ops callbacks")
> >>>>     4136ce90f079 ("ARM/dma-mapping: merge IOMMU ops")
> >>>> in reverse order, of course:
> >>>>     git revert 4136ce90f079
> >>>>     git revert d563bccfa35b
> >>>>     git revert 42998ef08aba
> >>>>     git revert ae626eb97376
> >>>>
> >>>> Christoph, Robin, since you are the authors of these commits, do you
> >>>> have any idea what could be happening? Are we able to fix this without
> >>>> reverting those commits, before 6.0?  
> >>>
> >>> "hangs shortly after init" isn't much to go on. Are any errors logged?
> >>> Possibly some driver is sat waiting for a DMA transfer to complete, that
> >>> has somehow got the wrong address or lost coherency so never gets seen,
> >>> but without at least being able to narrow it down to the affected driver
> >>> it's hard to do much more than vague guessing.  
> >>
> >> OK I enabled CONFIG_DMA_API_DEBUG and now am getting a null pointer
> >> dereference. I managed to isolate the bug to a specifc line in mvneta
> >> driver:
> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/mvneta.c#n2591
> >>
> >> I put debug printfs (pr_err("  a %i\n", __LINE__)) into the
> >> mvneta_rx_hwbm() function.
> >> The pr_err after the call to dma_sync_single_range_for_cpu() prints,
> >> but the pr_err after skb_put_data() does not print.
> >>
> >> Attaching console output.  
> > 
> > It seems that the null pointer dereference comes from the data variable
> > having zero value. We assign
> >    data = (u8 *)(uintptr_t)rx_desc->buf_cookie;
> > rx_desc is obtained with function
> >    mvneta_rxq_next_desc_get()
> > 
> > rx queues are allocated in mvneta_rxq_sw_init() with
> > 
> >    /* Allocate memory for RX descriptors */
> >    rxq->descs = dma_alloc_coherent(pp->dev->dev.parent,
> > 				  rxq->size * MVNETA_DESC_ALIGNED_SIZE,
> > 				  &rxq->descs_phys, GFP_KERNEL);  
> 
> Hmm, making sense of that driver is beyond me at this time on a Friday 
> afternoon, and I can't tell whether this is immediately related, but:
> 
> [   10.406446] Register r5 information: 0-page vmalloc region starting 
> at 0xf10f3000 allocated at dma_common_contiguous_remap+0x68/0x84
> 
> definitely smells suspicious in its own right. Remapping 0 pages is bad 
> enough, but I'm also slightly wondering about remapping DMA allocations 
> at all - IIUC this is one of the mvebu SoCs where everything gets made 
> coherent by a bus notifier, so I wouldn't expect remaps except for 
> highmem, but the upstream DT suggests you probably don't have masses of 
> RAM either :/

Are the patches that cause the regression supposed to do only code
refactoring (although major), or are they supposed to be functional
changes?

Marek