[PATCH v3 2/2] arm64: Work around missing `bti c` in modules

[D Scott Phillips]

GCC does not insert a `bti c` instruction at the beginning of a function
when all callers reach the function through a direct branch[1]. In the case
of cross-section calls (like __init to non __init), a thunk may be inserted
which uses an indirect branch. If that happens, the first instruction in
the callee function will result in a Branch Target Exception due to the
missing `bti c`.

Handle Branch Target Exceptions which happen in the kernel due to module
calls from __init to non-__init by clearing PSTATE.BTYPE and resuming.

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671

Signed-off-by: D Scott Phillips <scott at os.amperecomputing.com>
---
 arch/arm64/Kconfig        |  3 +++
 arch/arm64/kernel/traps.c | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9fb9fff08c94..8038842fa6b9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1896,6 +1896,9 @@ config ARM64_BTI_KERNEL
 	  is enabled and the system supports BTI all kernel code including
 	  modular code must have BTI enabled.
 
+config CC_HAS_CROSS_SECTION_BTI_MISSING
+	def_bool CC_IS_GCC # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671
+
 config CC_HAS_BRANCH_PROT_PAC_RET_BTI
 	# GCC 9 or later, clang 8 or later
 	def_bool $(cc-option,-mbranch-protection=pac-ret+leaf+bti)
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 56e1782fcf54..315a305a4f1d 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -501,11 +501,46 @@ NOKPROBE_SYMBOL(do_undefinstr);
 
 void do_bti(struct pt_regs *regs)
 {
+	struct module *mod;
+
 	if (user_mode(regs)) {
 		force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
 		return;
 	}
 
+	/*
+	 * GCC does not insert a `bti c` instruction at the beginning
+	 * of a function when all callers reach the function through a
+	 * direct branch. In the case of cross-section calls (like
+	 * __init to non __init), a thunk may be inserted which uses
+	 * an indirect branch. If that happens, the first instruction
+	 * in the callee function will result in a Branch Target
+	 * Exception due to the missing `bti c`.
+	 *
+	 * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671
+	 *
+	 * If that's the case here, clear PSTATE.BTYPE and resume.
+	 */
+	if (IS_ENABLED(CONFIG_CC_HAS_CROSS_SECTION_BTI_MISSING)) {
+		preempt_disable();
+		mod = __module_text_address(regs->pc);
+
+		if (mod && try_module_get(mod)) {
+			bool from_init;
+
+			from_init = within_module_init(regs->regs[30], mod);
+			module_put(mod);
+
+			if (from_init) {
+				preempt_enable();
+				regs->pstate &= ~PSR_BTYPE_MASK;
+				return;
+			}
+		}
+
+		preempt_enable();
+	}
+
 	die("Oops - BTI", regs, 0);
 }
 NOKPROBE_SYMBOL(do_bti);
-- 
2.37.2