[PATCH 4/5 v16] ARM: Initialize the mapping of KASan shadow memory

[Mike Rapoport]

On Mon, Oct 19, 2020 at 10:41:39AM +0200, Linus Walleij wrote:
> This patch initializes KASan shadow region's page table and memory.
> There are two stage for KASan initializing:
> 
> 1. At early boot stage the whole shadow region is mapped to just
>    one physical page (kasan_zero_page). It is finished by the function
>    kasan_early_init which is called by __mmap_switched(arch/arm/kernel/
>    head-common.S)
> 
> 2. After the calling of paging_init, we use kasan_zero_page as zero
>    shadow for some memory that KASan does not need to track, and we
>    allocate a new shadow space for the other memory that KASan need to
>    track. These issues are finished by the function kasan_init which is
>    call by setup_arch.
> 
> When using KASan we also need to increase the THREAD_SIZE_ORDER
> from 1 to 2 as the extra calls for shadow memory uses quite a bit
> of stack.
> 
> As we need to make a temporary copy of the PGD when setting up
> shadow memory we create a helpful PGD_SIZE definition for both
> LPAE and non-LPAE setups.
> 
> The KASan core code unconditionally calls pud_populate() so this
> needs to be changed from BUG() to do {} while (0) when building
> with KASan enabled.
> 
> After the initial development by Andre Ryabinin several modifications
> have been made to this code:
> 
> Abbott Liu <liuwenliang at huawei.com>
> - Add support ARM LPAE: If LPAE is enabled, KASan shadow region's
>   mapping table need be copied in the pgd_alloc() function.
> - Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate,
>   kasan_pgd_populate from .meminit.text section to .init.text section.
>   Reported by Florian Fainelli <f.fainelli at gmail.com>
> 
> Linus Walleij <linus.walleij at linaro.org>:
> - Drop the custom mainpulation of TTBR0 and just use
>   cpu_switch_mm() to switch the pgd table.
> - Adopt to handle 4th level page tabel folding.
> - Rewrite the entire page directory and page entry initialization
>   sequence to be recursive based on ARM64:s kasan_init.c.
> 
> Ard Biesheuvel <ardb at kernel.org>:
> - Necessary underlying fixes.
> - Crucial bug fixes to the memory set-up code.
> 
> Cc: Alexander Potapenko <glider at google.com>
> Cc: Dmitry Vyukov <dvyukov at google.com>
> Cc: kasan-dev at googlegroups.com
> Cc: Mike Rapoport <rppt at linux.ibm.com>
> Co-developed-by: Andrey Ryabinin <aryabinin at virtuozzo.com>
> Co-developed-by: Abbott Liu <liuwenliang at huawei.com>
> Co-developed-by: Ard Biesheuvel <ardb at kernel.org>
> Acked-by: Mike Rapoport <rppt at linux.ibm.com>
> Reviewed-by: Ard Biesheuvel <ardb at kernel.org>
> Tested-by: Ard Biesheuvel <ardb at kernel.org> # QEMU/KVM/mach-virt/LPAE/8G
> Tested-by: Florian Fainelli <f.fainelli at gmail.com> # Brahma SoCs
> Tested-by: Ahmad Fatoum <a.fatoum at pengutronix.de> # i.MX6Q
> Reported-by: Russell King - ARM Linux <linux at armlinux.org.uk>
> Reported-by: Florian Fainelli <f.fainelli at gmail.com>
> Signed-off-by: Andrey Ryabinin <aryabinin at virtuozzo.com>
> Signed-off-by: Abbott Liu <liuwenliang at huawei.com>
> Signed-off-by: Florian Fainelli <f.fainelli at gmail.com>
> Signed-off-by: Ard Biesheuvel <ardb at kernel.org>
> Signed-off-by: Linus Walleij <linus.walleij at linaro.org>
> ---

...

> +	cpu_switch_mm(tmp_pgd_table, &init_mm);
> +	local_flush_tlb_all();
> +
> +	clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END);
> +
> +	kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START),
> +				    kasan_mem_to_shadow((void *)-1UL) + 1);
> +
> +	for_each_memblock(memory, reg) {
> +		void *start = __va(reg->base);
> +		void *end = __va(reg->base + reg->size);
> +

I've killed for_each_memblock() recently and we have now 

	for_each_mem_range(idx, &pa_start, &pa_end)

instead.

> +		/* Do not attempt to shadow highmem */
> +		if (reg->base >= arm_lowmem_limit) {
> +			pr_info("Skip highmem block %pap-%pap\n",
> +				&reg->base, &reg->base + reg->size);
> +			continue;
> +		}
> +		if (reg->base + reg->size > arm_lowmem_limit) {
> +			pr_info("Truncating shadow for %pap-%pap to lowmem region\n",
> +				&reg->base, &reg->base + reg->size);
> +			end = __va(arm_lowmem_limit);
> +		}
> +		if (start >= end) {
> +			pr_info("Skipping invalid memory block %px-%px\n",
> +				start, end);
> +			continue;
> +		}
> +
> +		create_mapping(start, end);
> +	}
> +
> +	/*
> +	 * 1. The module global variables are in MODULES_VADDR ~ MODULES_END,
> +	 *    so we need to map this area.
> +	 * 2. PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR
> +	 *    ~ MODULES_END's shadow is in the same PMD_SIZE, so we can't
> +	 *    use kasan_populate_zero_shadow.
> +	 */
> +	create_mapping((void *)MODULES_VADDR, (void *)(PKMAP_BASE + PMD_SIZE));
> +
> +	/*
> +	 * KAsan may reuse the contents of kasan_early_shadow_pte directly, so
> +	 * we should make sure that it maps the zero page read-only.
> +	 */
> +	for (i = 0; i < PTRS_PER_PTE; i++)
> +		set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE,
> +			   &kasan_early_shadow_pte[i],
> +			   pfn_pte(virt_to_pfn(kasan_early_shadow_page),
> +				__pgprot(pgprot_val(PAGE_KERNEL)
> +					 | L_PTE_RDONLY)));
> +
> +	cpu_switch_mm(swapper_pg_dir, &init_mm);
> +	local_flush_tlb_all();
> +
> +	memset(kasan_early_shadow_page, 0, PAGE_SIZE);
> +	pr_info("Kernel address sanitizer initialized\n");
> +	init_task.kasan_depth = 0;
> +}
> diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c
> index c5e1b27046a8..f8e9bc58a84f 100644
> --- a/arch/arm/mm/pgd.c
> +++ b/arch/arm/mm/pgd.c
> @@ -66,7 +66,21 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
>  	new_pmd = pmd_alloc(mm, new_pud, 0);
>  	if (!new_pmd)
>  		goto no_pmd;
> -#endif
> +#ifdef CONFIG_KASAN
> +	/*
> +	 * Copy PMD table for KASAN shadow mappings.
> +	 */
> +	init_pgd = pgd_offset_k(TASK_SIZE);
> +	init_p4d = p4d_offset(init_pgd, TASK_SIZE);
> +	init_pud = pud_offset(init_p4d, TASK_SIZE);
> +	init_pmd = pmd_offset(init_pud, TASK_SIZE);
> +	new_pmd = pmd_offset(new_pud, TASK_SIZE);
> +	memcpy(new_pmd, init_pmd,
> +	       (pmd_index(MODULES_VADDR) - pmd_index(TASK_SIZE))
> +	       * sizeof(pmd_t));
> +	clean_dcache_area(new_pmd, PTRS_PER_PMD * sizeof(pmd_t));
> +#endif /* CONFIG_KASAN */
> +#endif /* CONFIG_LPAE */
>  
>  	if (!vectors_high()) {
>  		/*
> -- 
> 2.26.2
> 

-- 
Sincerely yours,
Mike.