[PATCH v2] arm64: implement support for static call trampolines

[Peter Zijlstra]

On Wed, Oct 28, 2020 at 07:41:14PM +0100, Ard Biesheuvel wrote:
> +/*
> + * The static call trampoline consists of one of the following sequences:
> + *
> + *      (A)           (B)           (C)           (D)           (E)
> + * 00: BTI  C        BTI  C        BTI  C        BTI  C        BTI  C
> + * 04: B    fn       NOP           NOP           NOP           NOP
> + * 08: RET           RET           ADRP X16, fn  ADRP X16, fn  ADRP X16, fn
> + * 0c: NOP           NOP           ADD  X16, fn  ADD  X16, fn  ADD  X16, fn
> + * 10:                             BR   X16      RET           NOP
> + * 14:                                                         ADRP X16, &fn
> + * 18:                                                         LDR  X16, [X16, &fn]
> + * 1c:                                                         BR   X16
> + *
> + * The architecture permits us to patch B instructions into NOPs or vice versa
> + * directly, but patching any other instruction sequence requires careful
> + * synchronization. Since branch targets may be out of range for ordinary
> + * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
> + * sequences in some cases, which complicates things considerably; since any
> + * sleeping tasks may have been preempted right in the middle of any of these
> + * sequences, we have to carefully transform one into the other, and ensure
> + * that it is safe to resume execution at any point in the sequence for tasks
> + * that have already executed part of it.
> + *
> + * So the rules are:
> + * - we start out with (A) or (B)
> + * - a branch within immediate range can always be patched in at offset 0x4;
> + * - sequence (A) can be turned into (B) for NULL branch targets;
> + * - a branch outside of immediate range can be patched using (C), but only if
> + *   . the sequence being updated is (A) or (B), or
> + *   . the branch target address modulo 4k results in the same ADD opcode
> + *     (which could occur when patching the same far target a second time)
> + * - once we have patched in (C) we cannot go back to (A) or (B), so patching
> + *   in a NULL target now requires sequence (D);
> + * - if we cannot patch a far target using (C), we fall back to sequence (E),
> + *   which loads the function pointer from memory.
> + *
> + * If we abide by these rules, then the following must hold for tasks that were
> + * interrupted halfway through execution of the trampoline:
> + * - when resuming at offset 0x8, we can only encounter a RET if (B) or (D)
> + *   was patched in at any point, and therefore a NULL target is valid;
> + * - when resuming at offset 0xc, we are executing the ADD opcode that is only
> + *   reachable via the preceding ADRP, and which is patched in only a single
> + *   time, and is therefore guaranteed to be consistent with the ADRP target;
> + * - when resuming at offset 0x10, X16 must refer to a valid target, since it
> + *   is only reachable via a ADRP/ADD pair that is guaranteed to be consistent.
> + *
> + * Note that sequence (E) is only used when switching between multiple far
> + * targets, and that it is not a terminal degraded state.
> + */

Would it make things easier if your trampoline consisted of two complete
slots, between which you can flip?

Something like:

	0x00	B 0x24 / NOP
	0x04    < slot 1 >
		....
	0x20
	0x24	< slot 2 >
		....
	0x40

Then each (20 byte) slot can contain any of the variants above and you
can write the unused slot without stop-machine. Then, when the unused
slot is populated, flip the initial instruction (like a static-branch),
issue synchronize_rcu_tasks() and flip to using the other slot for next
time.

Alternatively, you can patch the call-sites to point to the alternative
trampoline slot, but that might be pushing things a bit.