bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Wednesday, July 29, 2015 10:48 AM
>> To: Yoder Stuart-B08248
>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> Subject: Re: bug in identity map for 4KB pages?
>>
>> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>> >> Sent: Wednesday, July 29, 2015 6:43 AM
>> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
>> >> Subject: Re: bug in identity map for 4KB pages?
>> >>
>> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder at freescale.com> wrote:
>> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>> >> > > in order to have an idmap that covered all of RAM.
>> >> > >
>> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>> >> > > 64KB pages, but do not for 4KB pages.
>> >> > >
>> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>> >> > >
>> >> >
>> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> >
>> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>> > 0x80_8000_0000.
>> >
>> >> > So the physical arrangement still requires more than the 39 bits of
>> >> > virtual address space you get with 3 levels, even if the ID map can
>> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>> >> > __phys_to_virt() just wraps to a positive number.
>> >>
>> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> >> problem.
>> >>
>> >> We cut memory which would fall below the start of the linear map in
>> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>> >>
>> >> It looks like we already have the logic for cutting memory beyond the
>> >> end of the linear map, so we should just need to override the limit.
>> >
>> > What is the effect of cutting that memory?  ...are we losing something?
>> >
>>
>> Yes, you will lose everything above 2 GB in the current configuration.
>
> Hmmm...so you really mean that a 16 GB system will have only 2GB
> total available for both kernel/user space?   That won't be workable
> as a long term solution.
>

I agree. So that essentially means that 3 level page table kernels are
unsuitable for such systems.

>> >> Stuart, does the below patch prevent the panic you see?
>> >
>> > I will try it and let you know...can try the suggested patch by
>> > Ard as well.
>> >
>>
>> Is it possible to load the kernel at 0x80_0000_0000? I think your
>> system will just boot fine in that case, and you will only lose the
>> lower 2 GB in that case.
>
> So, it's like that lower 2 GB does not exist at all? ...that is going to
> be hard for our customers to swallow I think.
>
> With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>

Indeed.

> If it's true that due to our physical memory map that we will not
> be able to access all RAM with a standard defconfig, then makes
> me wonder whether we need to request our hardware designers to change
> that physical map.  (but, IIRC this physical map is that recommended
> by ARM)
>

Well, I am aware of an ARM document that explains the placement at
0x8000_0000, 0x80_8000_0000 etc.
But as far as I know, this is not a general recommendation for
hardware, but an internal guideline for the reference platforms. Of
course, you need some memory below 4 GB if you want to boot 32-bit
kernels without jumping through a lot of hoops, i.e., you won't be
able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
memory at 0x80_0000_0000

>> Re patch: I am working on a more comprehensive patch atm. Could you
>> perhaps try that instead (~15 min)?
>
> Sure, will wait for your patch and will try it.
>

Cheers,
Ard.