bug in identity map for 4KB pages?

[Stuart Yoder]

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Wednesday, July 29, 2015 11:12 AM
> To: Yoder Stuart-B08248
> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: bug in identity map for 4KB pages?
> 
> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder at freescale.com> wrote:
> >
> >> -----Original Message-----
> >> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> Sent: Wednesday, July 29, 2015 10:48 AM
> >> To: Yoder Stuart-B08248
> >> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> >> kernel at lists.infradead.org
> >> Subject: Re: bug in identity map for 4KB pages?
> >>
> >> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder at freescale.com> wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
> >> >> Sent: Wednesday, July 29, 2015 6:43 AM
> >> >> To: Ard Biesheuvel; Yoder Stuart-B08248
> >> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> >> >> Subject: Re: bug in identity map for 4KB pages?
> >> >>
> >> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder at freescale.com> wrote:
> >> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
> >> >> > > in order to have an idmap that covered all of RAM.
> >> >> > >
> >> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
> >> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
> >> >> > > 64KB pages, but do not for 4KB pages.
> >> >> > >
> >> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
> >> >> > >
> >> >> >
> >> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >> >
> >> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
> >> > 0x80_8000_0000.
> >> >
> >> >> > So the physical arrangement still requires more than the 39 bits of
> >> >> > virtual address space you get with 3 levels, even if the ID map can
> >> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
> >> >> > __phys_to_virt() just wraps to a positive number.
> >> >>
> >> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
> >> >> problem.
> >> >>
> >> >> We cut memory which would fall below the start of the linear map in
> >> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >> >>
> >> >> It looks like we already have the logic for cutting memory beyond the
> >> >> end of the linear map, so we should just need to override the limit.
> >> >
> >> > What is the effect of cutting that memory?  ...are we losing something?
> >> >
> >>
> >> Yes, you will lose everything above 2 GB in the current configuration.
> >
> > Hmmm...so you really mean that a 16 GB system will have only 2GB
> > total available for both kernel/user space?   That won't be workable
> > as a long term solution.
> >
> 
> I agree. So that essentially means that 3 level page table kernels are
> unsuitable for such systems.
> 
> >> >> Stuart, does the below patch prevent the panic you see?
> >> >
> >> > I will try it and let you know...can try the suggested patch by
> >> > Ard as well.
> >> >
> >>
> >> Is it possible to load the kernel at 0x80_0000_0000? I think your
> >> system will just boot fine in that case, and you will only lose the
> >> lower 2 GB in that case.
> >
> > So, it's like that lower 2 GB does not exist at all? ...that is going to
> > be hard for our customers to swallow I think.
> >
> > With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
> >
> 
> Indeed.
> 
> > If it's true that due to our physical memory map that we will not
> > be able to access all RAM with a standard defconfig, then makes
> > me wonder whether we need to request our hardware designers to change
> > that physical map.  (but, IIRC this physical map is that recommended
> > by ARM)
> >
> 
> Well, I am aware of an ARM document that explains the placement at
> 0x8000_0000, 0x80_8000_0000 etc.
> But as far as I know, this is not a general recommendation for
> hardware, but an internal guideline for the reference platforms. Of
> course, you need some memory below 4 GB if you want to boot 32-bit
> kernels without jumping through a lot of hoops, i.e., you won't be
> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
> memory at 0x80_0000_0000
> 
> >> Re patch: I am working on a more comprehensive patch atm. Could you
> >> perhaps try that instead (~15 min)?
> >
> > Sure, will wait for your patch and will try it.

Thanks. Tried your patch and the kernel boots (leaving 2GB of memory available).

[cut]
earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
bootconsole [uart0] enabled
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
Ignoring memory outside of linear range (0x004080000000 - 0x0083dfffffff)
cma: Reserved 16 MiB at 0x00000000fec00000
[cut]

So, this still leaves us unable to access all of memory without
48-bit VA or 64KB pages (neither of which are on by default).
(I had started a thread on 48-bit VA last week, and will continue
the discussion there)

Stuart