[PATCH 2/2] lib: sbi: Clear IPIs before init_warm_startup in non-boot harts

[Evgenii Shatokhin]

Since commit 50d4fde1c5a4 ("lib: Remove redundant sbi_platform_ipi_clear()
calls"), the IPI sent from the boot hart in wake_coldboot_harts() is not
cleared in the secondary harts until they reach sbi_ipi_init(). However,
sbi_hsm_init() and sbi_hsm_hart_wait() are called earlier, so a secondary
hart might enter sbi_hsm_hart_wait() with an already pending IPI.

Because of this, the secondary hart might exit the "wfi" loop in
sbi_hsm_hart_wait() right after the boot hart changes state of that hart to
SBI_HSM_STATE_START_PENDING. Then, depending on the hardware, it might be
able to reach sbi_hart_switch_mode() at the end of init_warmboot() before
the boot hart has set next_addr, etc., for it in sbi_hsm_hart_start(). In
this case, the secondary hart will jump to a wrong address.

This behaviour can be observed, for example, in a QEMU VM (QEMU 7.2.0) with
"-machine virt" running a Linux guest. Inserting delays after setting the
hart state to SBI_HSM_STATE_START_PENDING in sbi_hsm_hart_start() allows
reproducing the issue more reliably.

The comment in wait_for_coldboot() suggests that the initial IPI is needed
in the warm resume path, so let us clear it before init_warm_startup()
only.

To do this, sbi_ipi_raw_clear() was created similar to sbi_ipi_raw_send().

Signed-off-by: Evgenii Shatokhin <e.shatokhin at yadro.com>
---
 include/sbi/sbi_ipi.h | 2 ++
 lib/sbi/sbi_init.c    | 6 ++++--
 lib/sbi/sbi_ipi.c     | 6 ++++++
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/sbi/sbi_ipi.h b/include/sbi/sbi_ipi.h
index f6ac807..f384e74 100644
--- a/include/sbi/sbi_ipi.h
+++ b/include/sbi/sbi_ipi.h
@@ -77,6 +77,8 @@ void sbi_ipi_process(void);
 
 int sbi_ipi_raw_send(u32 target_hart);
 
+void sbi_ipi_raw_clear(u32 target_hart);
+
 const struct sbi_ipi_device *sbi_ipi_get_device(void);
 
 void sbi_ipi_set_device(const struct sbi_ipi_device *dev);
diff --git a/lib/sbi/sbi_init.c b/lib/sbi/sbi_init.c
index e353c33..a9a4084 100644
--- a/lib/sbi/sbi_init.c
+++ b/lib/sbi/sbi_init.c
@@ -438,10 +438,12 @@ static void __noreturn init_warmboot(struct sbi_scratch *scratch, u32 hartid)
 	if (hstate < 0)
 		sbi_hart_hang();
 
-	if (hstate == SBI_HSM_STATE_SUSPENDED)
+	if (hstate == SBI_HSM_STATE_SUSPENDED) {
 		init_warm_resume(scratch);
-	else
+	} else {
+		sbi_ipi_raw_clear(hartid);
 		init_warm_startup(scratch, hartid);
+	}
 
 	sbi_hart_switch_mode(hartid, scratch->next_arg1,
 			     scratch->next_addr,
diff --git a/lib/sbi/sbi_ipi.c b/lib/sbi/sbi_ipi.c
index 1bcc2e4..b9f6205 100644
--- a/lib/sbi/sbi_ipi.c
+++ b/lib/sbi/sbi_ipi.c
@@ -217,6 +217,12 @@ int sbi_ipi_raw_send(u32 target_hart)
 	return 0;
 }
 
+void sbi_ipi_raw_clear(u32 target_hart)
+{
+	if (ipi_dev && ipi_dev->ipi_clear)
+		ipi_dev->ipi_clear(target_hart);
+}
+
 const struct sbi_ipi_device *sbi_ipi_get_device(void)
 {
 	return ipi_dev;
-- 
2.34.1