Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi Palmer,

Thanks for your answer.

On 2026-09-17 12:18, Palmer Dabbelt wrote:
> On Tue, 15 Sep 2026 14:14:10 PDT (-0700), aurelien at aurel32.net wrote:
> > Hi,
> > 
> > Some more update, even if the progress is quite low.
> 
> Well, thanks for digging into this.
> 
> > 
> > On 2026-09-09 18:45, Aurelien Jarno wrote:
> > > [Added Andy and Karl in Cc: as they have been involved in the
> > > vectored user copy code and fighting similar issues]
> > > 
> > > Hi,
> > > 
> > > Some more progress on that topic.
> > > 
> > > On 2026-09-08 06:42, Aurelien Jarno wrote:
> > > > Dear all,
> > > > > I have done some small progress on that issue. Help is still
> > > wanted and > would be appreciated.
> > > > > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > > > Dear all,
> > > > > > > For the last weeks, I have been tracking a random memory
> > > corruption and
> > > > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V
> > > Jupiter). It > > started upgrading to glibc 2.43, which does
> > > memset() through vector > > instructions.  It is reproducible using
> > > the Debian 7.1.7-1~bpo13+1 > > kernel, but I have also been able to
> > > reproduce it with a vanilla 7.2.2 > > kernel, using a similar
> > > configuration to the Debian kernel. The board > > uses OpenSBI 1.9
> > > and the vendor U-Boot.
> > 
> > I have been able to also reproduce the issue with the vendor OpenSBI. It
> > is also reproducible with linux 7.2.6.
> > 
> > > > I have been able to rule out OpenSBI from the issue, I have
> > > checked > there is not trap to OpenSBI when the problem happens.
> > > > > > Typically it manifests itself with the following kind of
> > > error, when > > running g++ from GCC 16 as part of building software
> > > (e.g. OpenJDK, > > Blender, Dolfin, Qt6):   > > > > Assembler
> > > messages:
> > > > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > > > > > The broken chars are 0xff and it seems there are always 240
> > > (but with > > poor statistics). Sometimes it instead causes a GCC
> > > ICE instead.
> > > > > I have identified that the 0xff comes from the poisoning done
> > > for v0 in
> > > > __riscv_v_vstate_discard(), which should only happen for a
> > > syscall. > Indeed changing the value to different ones propagates to
> > > the above > error message.
> > > > > Furthermore I have found that changing
> > > CONFIG_RISCV_ISA_V_PREEMPTIVE > doesn't change anything, and that
> > > changing RISCV_ISA_V_UCOPY_THRESHOLD > changes the number of broken
> > > bytes.
> > > 
> > > The values from __riscv_v_vstate_discard() end-up there because they
> > > are the last values written to the vector registers. I have added
> > > some poisoning in __asm_vector_usercopy_sum_enabled before the loop,
> > > and those values appear instead. Using different values per vector
> > > register, I have found that the corruption comes from a partial load
> > > of the vle8.v instruction, while the result of the poisoning and the
> > > partial load are
> > > then both written by the vse8.v:
> > > 
> > >  loop:
> > >         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
> > >         fixup vle8.v vData, (pSrc), 10f
> > >         sub iNum, iNum, iVL
> > >         add pSrc, pSrc, iVL
> > >         fixup vse8.v vData, (pDst), 11f
> > >         add pDst, pDst, iVL
> > >         bnez iNum, loop
> > 
> > I have instrumented that code, basically adding a function to check the
> > copy afterwards (only using scalar instructions), and report values in
> > case of errors. So far I have been able to catch the issue 4 times that
> > way (including one where I haven't seen any visible consequence on the
> > userland).
> > 
> > This is what I observed:
> > - The issue always happens for the first bytes of the user copy, which
> > in all cases were supposed to be up to 4096 bytes long.
> > - On the first iteration of the loop only the first 16 bytes are loaded
> > by the vle8.v instruction but the full 256 bytes are written to the
> > destination by the vse8.v instrution.
> 
> That sounds like some sort of trap handling issue: something's filling out
> the first word of the vector, but then not the rest, but then not triggering
> a restart of the instruction.  We talked some on IRC and it sounds like that

Yes, it's really looks like the instruction is interrupted but not 
restarted, instead the following instruction is executed.

> trap isn't getting into Linux, so it's possible something in M-mode is
> mangling it -- this is where we start to get into vstart/idempotent

If you mean sbi_misaligned_v_ld_emulator() and 
sbi_misaligned_v_st_emulator(), I have already ruled the mout. They are 
not called because both userland (glibc 2.43) and kernel (vector user 
copy) are using e8, so there are no possible alignment issues.


> instruction territory and that's not well tested by running on QEMU, so it
> wouldn't be super surprising we're just stumbling into a bug there.

I don't think it's a vstart issue. If vstart were wrongly restored to 0, 
the whole vector would be loaded again, inefficient but correct. If 
vstart were restored to another wrong value, the end of vector would 
also be loaded, overriding the values from riscv_v_vstate_discard(). The 
only vstart case that could explain that is vstart >= vl, but I don't 
think this is allowed.

> I have a K3 that I've been meaning to play around with, I'm going to go try
> to see if I can get it running and reproduce this one.

Just to be clear, I only observed corruption in the vectored user copy 
code on the K1. But the K3 is affected by similar random corruption than 
the K1 that disappear when disabling the vector unit. I believe that the 
corruption happens at multiple places, and that by chance on the K1 it 
can be observed on the vectored user copy.

To reproduce the issue, I use on a Debian installation (or on the 
original Bianbu installation) the following commands in a loop:
- sbuild -d sid blender
- sbuild -d sid --build-dep-resolver=aptitude --extra-repository="deb http://deb.debian.org/debian experimental main" --add-depends "libc6 (>= 2.44)" blender

The second command uses glibc 2.44, which uses more vector instructions 
via ifunc, so it crashes faster. Still it can takes a few hours until it 
happens. On the K1 it takes on average 5 hours, sometimes it takes up to 
18 hours before I can reproduce the issue.

> > - On the second iteration of the loop, the vle8.v causes a trap, which
> > ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20,
> > vstart=0x0 and vtype=0xc3.
> > - Considering the 2 first iterations of the loop (i.e. 512 bytes),
> > neither the source nor destination addresses cross a page.
> > 
> > All that said, it is what I can *observe*. I have no idea if the first
> > iteration of the loop also goes through fixup or not. Also in addition
> > to precise traps (which definitely happen for the second iteration), IRQ
> > could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my
> > test, in order to limit the code paths to review, so during that copy
> > preempt is disabled, but IRQ is not.

I tried disabling the IRQ around the vectored user copy, but it didn't 
change anything.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien at aurel32.net                     http://aurel32.net