REGRESSION in 6.0-rc7 caused by patch "ARM/dma-mapping: use dma-direct unconditionally"

[Robin Murphy]

On 2022-09-30 16:02, Marek Behún wrote:
> On Fri, 30 Sep 2022 16:52:34 +0200
> Marek Behún <kabel at kernel.org> wrote:
> 
>> On Fri, 30 Sep 2022 14:46:06 +0100
>> Robin Murphy <robin.murphy at arm.com> wrote:
>>
>>> On 2022-09-30 14:10, Marek Behún wrote:
>>>> Hello Linus, Arnd, Robin and Christoph,
>>>>
>>>> I just bisected a regression on Turris Omnia (Armada 385), wherein the
>>>> system hangs shortly after init is run, to commit
>>>>
>>>>     ae626eb97376 ("ARM/dma-mapping: use dma-direct unconditionally")
>>>>     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ae626eb97376
>>>>
>>>> In order to fix the regression, I had to revert this commit and
>>>> subsequent 3 commits:
>>>>     ae626eb97376 ("ARM/dma-mapping: use dma-direct unconditionally")
>>>>     42998ef08aba ("ARM/dma-mapping: drop .dma_supported for IOMMU ops")
>>>>     d563bccfa35b ("ARM/dma-mapping: consolidate IOMMU ops callbacks")
>>>>     4136ce90f079 ("ARM/dma-mapping: merge IOMMU ops")
>>>> in reverse order, of course:
>>>>     git revert 4136ce90f079
>>>>     git revert d563bccfa35b
>>>>     git revert 42998ef08aba
>>>>     git revert ae626eb97376
>>>>
>>>> Christoph, Robin, since you are the authors of these commits, do you
>>>> have any idea what could be happening? Are we able to fix this without
>>>> reverting those commits, before 6.0?
>>>
>>> "hangs shortly after init" isn't much to go on. Are any errors logged?
>>> Possibly some driver is sat waiting for a DMA transfer to complete, that
>>> has somehow got the wrong address or lost coherency so never gets seen,
>>> but without at least being able to narrow it down to the affected driver
>>> it's hard to do much more than vague guessing.
>>
>> OK I enabled CONFIG_DMA_API_DEBUG and now am getting a null pointer
>> dereference. I managed to isolate the bug to a specifc line in mvneta
>> driver:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/mvneta.c#n2591
>>
>> I put debug printfs (pr_err("  a %i\n", __LINE__)) into the
>> mvneta_rx_hwbm() function.
>> The pr_err after the call to dma_sync_single_range_for_cpu() prints,
>> but the pr_err after skb_put_data() does not print.
>>
>> Attaching console output.
> 
> It seems that the null pointer dereference comes from the data variable
> having zero value. We assign
>    data = (u8 *)(uintptr_t)rx_desc->buf_cookie;
> rx_desc is obtained with function
>    mvneta_rxq_next_desc_get()
> 
> rx queues are allocated in mvneta_rxq_sw_init() with
> 
>    /* Allocate memory for RX descriptors */
>    rxq->descs = dma_alloc_coherent(pp->dev->dev.parent,
> 				  rxq->size * MVNETA_DESC_ALIGNED_SIZE,
> 				  &rxq->descs_phys, GFP_KERNEL);

Hmm, making sense of that driver is beyond me at this time on a Friday 
afternoon, and I can't tell whether this is immediately related, but:

[   10.406446] Register r5 information: 0-page vmalloc region starting 
at 0xf10f3000 allocated at dma_common_contiguous_remap+0x68/0x84

definitely smells suspicious in its own right. Remapping 0 pages is bad 
enough, but I'm also slightly wondering about remapping DMA allocations 
at all - IIUC this is one of the mvebu SoCs where everything gets made 
coherent by a bus notifier, so I wouldn't expect remaps except for 
highmem, but the upstream DT suggests you probably don't have masses of 
RAM either :/

Thanks,
Robin.