Random corruption on SpacemiT K1 (and K3) with RVV

[Palmer Dabbelt]

On Tue, 15 Sep 2026 14:14:10 PDT (-0700), aurelien at aurel32.net wrote:
> Hi,
>
> Some more update, even if the progress is quite low.

Well, thanks for digging into this.

>
> On 2026-09-09 18:45, Aurelien Jarno wrote:
>> [Added Andy and Karl in Cc: as they have been involved in the vectored 
>> user copy code and fighting similar issues]
>> 
>> Hi,
>> 
>> Some more progress on that topic.
>> 
>> On 2026-09-08 06:42, Aurelien Jarno wrote:
>> > Dear all,
>> > 
>> > I have done some small progress on that issue. Help is still wanted and 
>> > would be appreciated.
>> > 
>> > On 2026-08-30 22:52, Aurelien Jarno wrote:
>> > > Dear all,
>> > > 
>> > > For the last weeks, I have been tracking a random memory corruption and
>> > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
>> > > started upgrading to glibc 2.43, which does memset() through vector 
>> > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
>> > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
>> > > kernel, using a similar configuration to the Debian kernel. The board 
>> > > uses OpenSBI 1.9 and the vendor U-Boot.
>
> I have been able to also reproduce the issue with the vendor OpenSBI. It 
> is also reproducible with linux 7.2.6.
>
>> > I have been able to rule out OpenSBI from the issue, I have checked 
>> > there is not trap to OpenSBI when the problem happens.
>> > 
>> > > Typically it manifests itself with the following kind of error, when 
>> > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
>> > > Blender, Dolfin, Qt6):   
>> > > 
>> > > Assembler messages:
>> > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
>> > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
>> > > 
>> > > The broken chars are 0xff and it seems there are always 240 (but with 
>> > > poor statistics). Sometimes it instead causes a GCC ICE instead.
>> > 
>> > I have identified that the 0xff comes from the poisoning done for v0 in
>> > __riscv_v_vstate_discard(), which should only happen for a syscall. 
>> > Indeed changing the value to different ones propagates to the above 
>> > error message.
>> > 
>> > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
>> > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
>> > changes the number of broken bytes.
>> 
>> The values from __riscv_v_vstate_discard() end-up there because they are 
>> the last values written to the vector registers. I have added some 
>> poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
>> those values appear instead. Using different values per vector register, 
>> I have found that the corruption comes from a partial load of the vle8.v 
>> instruction, while the result of the poisoning and the partial load are
>> then both written by the vse8.v:
>> 
>>  loop:
>>         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
>>         fixup vle8.v vData, (pSrc), 10f
>>         sub iNum, iNum, iVL
>>         add pSrc, pSrc, iVL
>>         fixup vse8.v vData, (pDst), 11f
>>         add pDst, pDst, iVL
>>         bnez iNum, loop
>
> I have instrumented that code, basically adding a function to check the 
> copy afterwards (only using scalar instructions), and report values in 
> case of errors. So far I have been able to catch the issue 4 times that 
> way (including one where I haven't seen any visible consequence on the 
> userland). 
>
> This is what I observed:
> - The issue always happens for the first bytes of the user copy, which 
>   in all cases were supposed to be up to 4096 bytes long.
> - On the first iteration of the loop only the first 16 bytes are loaded 
>   by the vle8.v instruction but the full 256 bytes are written to the 
>   destination by the vse8.v instrution.

That sounds like some sort of trap handling issue: something's filling 
out the first word of the vector, but then not the rest, but then not 
triggering a restart of the instruction.  We talked some on IRC and it 
sounds like that trap isn't getting into Linux, so it's possible 
something in M-mode is mangling it -- this is where we start to get into 
vstart/idempotent instruction territory and that's not well tested by 
running on QEMU, so it wouldn't be super surprising we're just stumbling 
into a bug there.

I have a K3 that I've been meaning to play around with, I'm going to go 
try to see if I can get it running and reproduce this one.

> - On the second iteration of the loop, the vle8.v causes a trap, which 
>   ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20, 
>   vstart=0x0 and vtype=0xc3.
> - Considering the 2 first iterations of the loop (i.e. 512 bytes), 
>   neither the source nor destination addresses cross a page.
>
> All that said, it is what I can *observe*. I have no idea if the first 
> iteration of the loop also goes through fixup or not. Also in addition 
> to precise traps (which definitely happen for the second iteration), IRQ 
> could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my 
> test, in order to limit the code paths to review, so during that copy 
> preempt is disabled, but IRQ is not.
>
> Regards
> Aurelien
>
> -- 
> Aurelien Jarno                          GPG: 4096R/1DDD8C9B
> aurelien at aurel32.net                     http://aurel32.net