[PATCH 7/7] riscv: mm: Pre-allocate vmalloc PGD leaves

[Björn Töpel]

From: Björn Töpel <bjorn at rivosinc.com>

Instead of relying on vmalloc_fault() to synchronize the page-tables,
pre-allocate the PGD leaves of the vmalloc area. This is only enabled
if memory hot/add is enabled by the build.

Signed-off-by: Björn Töpel <bjorn at rivosinc.com>
---
 arch/riscv/mm/fault.c | 7 ++++++-
 arch/riscv/mm/init.c  | 1 +
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 8685f85a7474..b61e279acd50 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -233,12 +233,17 @@ void handle_page_fault(struct pt_regs *regs)
 	 * Fault-in kernel-space virtual memory on-demand.
 	 * The 'reference' page table is init_mm.pgd.
 	 *
+	 * For memory hotplug enabled systems, the PGD entries are
+	 * pre-allocated, which avoids the need to synchronize
+	 * pgd/fault-in.
+	 *
 	 * NOTE! We MUST NOT take any locks for this case. We may
 	 * be in an interrupt or a critical region, and should
 	 * only copy the information from the master page table,
 	 * nothing more.
 	 */
-	if (unlikely((addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
+	if (unlikely(!IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
+		     (addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
 		vmalloc_fault(regs, code, addr);
 		return;
 	}
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index a468708d1e1c..fd5a6d3fe182 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -236,6 +236,7 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon
 static void __init prepare_memory_hotplug(void)
 {
 #ifdef CONFIG_MEMORY_HOTPLUG
+	preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
 	preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
 	preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
 #endif
-- 
2.39.2