[ARM]Fixmap mapping region is not enough for system of 14+ CPUs.

[Nicolas Pitre]

On Thu, 13 Mar 2014, Liu hua wrote:

> On 2014/3/13 1:08, Nicolas Pitre wrote:
> > On Wed, 12 Mar 2014, Liu hua wrote:
> > 
> >> Hi Russell, Will or Nicolas,
> >>
> >> (In this mail, we only discuss ARM 32-bit linux.)
> >>
> >> As we know, the region (0xfff00000-0xfffdffff) is reserved as fixmap
> >> mapping region.
> >>
> >> The function "kmap_atomic" maps highmem pages to this region referring
> >> to CPUID and per-cpu variable "__kmap_atomic_idx" via
> >>
> >>   idx = type + KM_TYPE_NR * smp_processor_id();
> >>   vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
> >>
> >> Size of region used by one cpu is 0x10000 (KM_TYPE_NR << PAGE_SHIFT).
> >> And the total size of the fixmap mapping region is 0xe0000.
> >> (only support 14 CPUs).
> >>
> >> So in a system of more than 14 CPUs, this region is not large enough.
> >> should we change the memory layout on ARM Linux to support 14+ cpu system ?
> >> Or can we do anything else to support that ?
> > 
> > How many CPUs do you have?
> > 
> > What about the following patch?  If this doesn't work for you then more 
> > intrusive changes will be needed.
> > 
> > diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
> > index 68ea615c2a..254f2df08d 100644
> > --- a/arch/arm/include/asm/fixmap.h
> > +++ b/arch/arm/include/asm/fixmap.h
> > @@ -14,7 +14,17 @@
> >   */
> >  
> >  #define FIXADDR_START		0xfff00000UL
> > +
> > +#if !defined(CONFIG_HAVE_TCM) && !defined(CONFIG_CPU_XSCALE)
> > +/*
> > + * If no TCM nor on on a XScale then enlarge the fixmap area to
> > + * accommodate up to 30 CPUs.
> > + */
> > +#define FIXADDR_END		0xffff0000UL
> > +#else
> >  #define FIXADDR_END		0xfffe0000UL
> One cpu need 0x10000-size region; So this patch can only support
> uo to 15 CPUs
Hmmm... You're right of course.

> We have 16 CPUs in our system. And your patch cannot work if
> the CPU number exceeds 15.
> 
> In addition, we can configure NR_CPUS up to 32. So should we
> solve this problem completely?
> 
> There is an 1M-size hole between DMA mapping region and fixmap
> mapping region(0xffe00000-0xfff00000).if this region are belonged
> to fixmap mapping region, it can only support 30 CPUs.
> 
> Since we have alloced a second level page table to coverring
> 0xffe00000 - 0xffffffff(2M). And 0xffff0000 upwards used as
> vector pages. So if the left region is all used as fixmap mapping
> region(size : 0x1f0000). Up to 31 CPUs is supported.
> 
> 
> The following patch can solve our problem(up to 30 CPUs), But it
> reduce the DMA region. What do you think about it ?
> Is is necessory to solve this problem completely(up to 32 CPUs)
> ?
> 
> 
> Liu Hua
> 
> -----------------
> diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h
> index 68ea615..8379891 100644
> --- a/arch/arm/include/asm/fixmap.h
> +++ b/arch/arm/include/asm/fixmap.h
> @@ -13,7 +13,7 @@
>   * 0xfffe0000 and 0xfffeffff.
>   */
> 
> -#define FIXADDR_START              0xfff00000UL
> +#define FIXADDR_START                0xffe00000UL
>  #define FIXADDR_END            0xfffe0000UL
>  #define FIXADDR_TOP            (FIXADDR_END - PAGE_SIZE)
> 
> diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
> index 4afb376..0c40674 100644
> --- a/arch/arm/include/asm/memory.h
> +++ b/arch/arm/include/asm/memory.h
> @@ -83,7 +83,7 @@
>   */
>  #define IOREMAP_MAX_ORDER      24
> 
> -#define CONSISTENT_END             (0xffe00000UL)
> +#define CONSISTENT_END               (0xffd00000UL)

Some systems might possibly have issues with a smaller DMA region. 
I don't recall which though.

you should also update Documentation/arm/memory.txt in your patch.
Then I'd suggest you can repost it more formally for comments.


Nicolas