Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi Andy,

On 2026-09-22 16:33, Andy Chiu wrote:
> Hi Aurelien,
> 
> Sorry for replying late. I did run a compilation test with vectorized
> glibc in qemu after seeing this thread shortly, but then focus on
> something else as it didn't catch any corruption we've seen here.

Thanks for your feedback. Note that at this stage I have not been able 
to reproduce the issue with QEMU. I guess it's very timing dependent, 
also I am not sure if QEMU simulates partially executed instructions 
(outside of page faults).

> On Wed, Sep 09, 2026 at 06:45:14PM +0200, Aurelien Jarno wrote:
> > [Added Andy and Karl in Cc: as they have been involved in the vectored 
> > user copy code and fighting similar issues]
> > 
> > Hi,
> > 
> > Some more progress on that topic.
> > 
> > On 2026-09-08 06:42, Aurelien Jarno wrote:
> > > Dear all,
> > > 
> > > I have done some small progress on that issue. Help is still wanted and 
> > > would be appreciated.
> > > 
> > > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > > Dear all,
> > > > 
> > > > For the last weeks, I have been tracking a random memory corruption and
> > > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > > > started upgrading to glibc 2.43, which does memset() through vector 
> > > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > > > kernel, using a similar configuration to the Debian kernel. The board 
> > > > uses OpenSBI 1.9 and the vendor U-Boot.
> 
> The KVM fix, which is in 7.3 now, is irrelavant to this bug (see below),
> because the fix targets preemptible kernel-mode vector, but we can
> trigger this bug with CONFIG_RISCV_ISA_V_PREEMPTIVE unset. Nontheless,
> if we want to test on the latest code, feel free to grab the series at
> [3] and boot with riscv_novstateopt to preseve the context poisoning
> behavior.

Thanks, I'll try that.

> > > 
> > > I have been able to rule out OpenSBI from the issue, I have checked 
> > > there is not trap top OpenSBI when the problem happens.
> > > 
> > > > Typically it manifests itself with the following kind of error, when 
> > > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > > > Blender, Dolfin, Qt6):   
> > > > 
> > > > Assembler messages:
> > > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > > 
> > > > The broken chars are 0xff and it seems there are always 240 (but with 
> > > > poor statistics). Sometimes it instead causes a GCC ICE instead.
> > > 
> > > I have identified that the 0xff comes from the poisoning done for v0 in
> > > __riscv_v_vstate_discard(), which should only happen for a syscall. 
> > > Indeed changing the value to different ones propagates to the above 
> > > error message.
> > > 
> > > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> > > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> > > changes the number of broken bytes.
> 
> Turnning off CONFIG_RISCV_ISA_V_PREEMPTIVE makes the ucopy fall back to
> the scalar copy on a page fault, because faulthandler_disabled() would
> return true after kernel_vector_begin(). So if we are suspecting a
> corruption at vector restart, then it is not, there is no restarting of
> such instruction.
> 
> The only possible way to have a restarted vector ld/st here under
> !CONFIG_RISCV_ISA_V_PREEMPTIVE is then an irq restart. I don't know if
> you test this code with !CONFIG_RISCV_ISA_V_PREEMPTIVE **and** with
> irqs_disabled() == true in the user copy code. If we still observed
> a corruption under this case, then it suggests that the corruption
> happens even eariler. It is then either the faulting instruction itself,
> or the corruption happens even eariler.

I have already tried with both CONFIG_RISCV_ISA_V_PREEMPTIVE=n and 
wrapping the vectored user copy call in between local_irq_save(flags) 
and local_irq_restore(flags). The problem is still reproducible.

I have also tried to instrument the vectored user copy code by saving 
the insret crs before and after the vle8.v instruction to detect a 
possible interrupted instruction (as suggested on IRC), but adding this 
code seems to have "fixed" the issue.

> > The values from __riscv_v_vstate_discard() end-up there because they are 
> > the last values written to the vector registers. I have added some 
> > poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
> > those values appear instead. Using different values per vector register, 
> 
> Do you know what is the address of corrupted data? If it always starts at
> a page boundary (e.g offset 0x00000 for THP) then it suggests a strong
> correlation with fault handling.

On the K1, for the few cases I captured, none of them involved 4k page 
crossing on either source or destination. The only invariant I have 
found is that for some reason the first iteration of the loop only 
loads 16 bytes, and write 256 bytes. The second iteration of the loop 
then goes to the load fixup and exits.

On the K3, I have never observed a corruption at the vector user copy, 
but I still observed rare memory corruptions causing gcc to crash, and 
which disappeared when disabling vector instructions at the device tree
level, like on the K1. But that could be two different issues linked to 
vector instructions.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien at aurel32.net                     http://aurel32.net