bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 18:12, Ard Biesheuvel <ard.biesheuvel at linaro.org> wrote:
> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>>
>>> -----Original Message-----
>>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>>> Sent: Wednesday, July 29, 2015 10:48 AM
>>> To: Yoder Stuart-B08248
>>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>>> kernel at lists.infradead.org
>>> Subject: Re: bug in identity map for 4KB pages?
>>>
>>> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>>> >
>>> >
>>> >> -----Original Message-----
>>> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>>> >> Sent: Wednesday, July 29, 2015 6:43 AM
>>> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>>> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
>>> >> Subject: Re: bug in identity map for 4KB pages?
>>> >>
>>> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>>> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>>> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>>> >> > > in order to have an idmap that covered all of RAM.
>>> >> > >
>>> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>>> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>>> >> > > 64KB pages, but do not for 4KB pages.
>>> >> > >
>>> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>>> >> > >
>>> >> >
>>> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>>> >
>>> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>>> > 0x80_8000_0000.
>>> >
>>> >> > So the physical arrangement still requires more than the 39 bits of
>>> >> > virtual address space you get with 3 levels, even if the ID map can
>>> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>>> >> > __phys_to_virt() just wraps to a positive number.
>>> >>
>>> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>>> >> problem.
>>> >>
>>> >> We cut memory which would fall below the start of the linear map in
>>> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>>> >>
>>> >> It looks like we already have the logic for cutting memory beyond the
>>> >> end of the linear map, so we should just need to override the limit.
>>> >
>>> > What is the effect of cutting that memory?  ...are we losing something?
>>> >
>>>
>>> Yes, you will lose everything above 2 GB in the current configuration.
>>
>> Hmmm...so you really mean that a 16 GB system will have only 2GB
>> total available for both kernel/user space?   That won't be workable
>> as a long term solution.
>>
>
> I agree. So that essentially means that 3 level page table kernels are
> unsuitable for such systems.
>
>>> >> Stuart, does the below patch prevent the panic you see?
>>> >
>>> > I will try it and let you know...can try the suggested patch by
>>> > Ard as well.
>>> >
>>>
>>> Is it possible to load the kernel at 0x80_0000_0000? I think your
>>> system will just boot fine in that case, and you will only lose the
>>> lower 2 GB in that case.
>>
>> So, it's like that lower 2 GB does not exist at all? ...that is going to
>> be hard for our customers to swallow I think.
>>
>> With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>>
>
> Indeed.
>
>> If it's true that due to our physical memory map that we will not
>> be able to access all RAM with a standard defconfig, then makes
>> me wonder whether we need to request our hardware designers to change
>> that physical map.  (but, IIRC this physical map is that recommended
>> by ARM)
>>
>
> Well, I am aware of an ARM document that explains the placement at
> 0x8000_0000, 0x80_8000_0000 etc.
> But as far as I know, this is not a general recommendation for
> hardware, but an internal guideline for the reference platforms. Of
> course, you need some memory below 4 GB if you want to boot 32-bit
> kernels without jumping through a lot of hoops, i.e., you won't be
> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
> memory at 0x80_0000_0000
>

I found the doc in question:
http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf

However, it recommends the following layout for up to 512 GB of DRAM:
2 GB at 0x8000_0000
30 GB at 0x8_8000_0000
480 GB at 0x88_0000_0000