bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 21:19, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Wednesday, July 29, 2015 11:12 AM
>> To: Yoder Stuart-B08248
>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> Subject: Re: bug in identity map for 4KB pages?
>>
>> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>> >
>> >> -----Original Message-----
>> >> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> >> Sent: Wednesday, July 29, 2015 10:48 AM
>> >> To: Yoder Stuart-B08248
>> >> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> >> kernel at lists.infradead.org
>> >> Subject: Re: bug in identity map for 4KB pages?
>> >>
>> >> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>> >> >> Sent: Wednesday, July 29, 2015 6:43 AM
>> >> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>> >> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> >> >> Subject: Re: bug in identity map for 4KB pages?
>> >> >>
>> >> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> >> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>> >> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>> >> >> > > in order to have an idmap that covered all of RAM.
>> >> >> > >
>> >> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>> >> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>> >> >> > > 64KB pages, but do not for 4KB pages.
>> >> >> > >
>> >> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>> >> >> > >
>> >> >> >
>> >> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> >> >
>> >> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>> >> > 0x80_8000_0000.
>> >> >
>> >> >> > So the physical arrangement still requires more than the 39 bits of
>> >> >> > virtual address space you get with 3 levels, even if the ID map can
>> >> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>> >> >> > __phys_to_virt() just wraps to a positive number.
>> >> >>
>> >> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> >> >> problem.
>> >> >>
>> >> >> We cut memory which would fall below the start of the linear map in
>> >> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>> >> >>
>> >> >> It looks like we already have the logic for cutting memory beyond the
>> >> >> end of the linear map, so we should just need to override the limit.
>> >> >
>> >> > What is the effect of cutting that memory?  ...are we losing something?
>> >> >
>> >>
>> >> Yes, you will lose everything above 2 GB in the current configuration.
>> >
>> > Hmmm...so you really mean that a 16 GB system will have only 2GB
>> > total available for both kernel/user space?   That won't be workable
>> > as a long term solution.
>> >
>>
>> I agree. So that essentially means that 3 level page table kernels are
>> unsuitable for such systems.
>>
>> >> >> Stuart, does the below patch prevent the panic you see?
>> >> >
>> >> > I will try it and let you know...can try the suggested patch by
>> >> > Ard as well.
>> >> >
>> >>
>> >> Is it possible to load the kernel at 0x80_0000_0000? I think your
>> >> system will just boot fine in that case, and you will only lose the
>> >> lower 2 GB in that case.
>> >
>> > So, it's like that lower 2 GB does not exist at all? ...that is going to
>> > be hard for our customers to swallow I think.
>> >
>> > With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>> >
>>
>> Indeed.
>>
>> > If it's true that due to our physical memory map that we will not
>> > be able to access all RAM with a standard defconfig, then makes
>> > me wonder whether we need to request our hardware designers to change
>> > that physical map.  (but, IIRC this physical map is that recommended
>> > by ARM)
>> >
>>
>> Well, I am aware of an ARM document that explains the placement at
>> 0x8000_0000, 0x80_8000_0000 etc.
>> But as far as I know, this is not a general recommendation for
>> hardware, but an internal guideline for the reference platforms. Of
>> course, you need some memory below 4 GB if you want to boot 32-bit
>> kernels without jumping through a lot of hoops, i.e., you won't be
>> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
>> memory at 0x80_0000_0000
>>
>> >> Re patch: I am working on a more comprehensive patch atm. Could you
>> >> perhaps try that instead (~15 min)?
>> >
>> > Sure, will wait for your patch and will try it.
>
> Thanks. Tried your patch and the kernel boots (leaving 2GB of memory available).
>

OK. Thanks for confirming.

-- 
Ard.


> [cut]
> earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
> bootconsole [uart0] enabled
> efi: Getting EFI parameters from FDT:
> efi: UEFI not found.
> Ignoring memory outside of linear range (0x004080000000 - 0x0083dfffffff)
> cma: Reserved 16 MiB at 0x00000000fec00000
> [cut]
>
> So, this still leaves us unable to access all of memory without
> 48-bit VA or 64KB pages (neither of which are on by default).
> (I had started a thread on 48-bit VA last week, and will continue
> the discussion there)
>
> Stuart