Cache problems in user space - kernel space communication

[Van Schaik, Carl]

Looks like you are mapping the same page cached to both kernel and user space (at different virtual addresses). With the VIVT cache, the two mappings will hash differently into the cache and access different cache lines. To successfully work like this on a VIVT cache you need to perform a lot of cache maintenance whenever switching from kernel to user. You would likely rather map the page uncached in both kernel and user to save all the effort.

--
Carl

> -----Original Message-----
> From: Bart Jonkers [mailto:jonkersbart at gmail.com]
> Sent: Tuesday, 14 February 2012 6:46 PM
> To: linux-arm-kernel at lists.infradead.org
> Subject: Cache problems in user space - kernel space communication
> 
> All,
> 
> I'm experiencing problems in our application where a driver and a user space
> application performs bi-directional communication through a memory
> buffer.
> 
> We are running a 2.6.35.14 kernel on an kirkwood based SoC (Marvell
> 88F6192):
> [    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE),
> cr=00053977
> [    0.000000] CPU: VIVT data cache, VIVT instruction cache
> 
> /root> cat /proc/cpuinfo
> Processor       : Feroceon 88FR131 rev 1 (v5l)
> BogoMIPS        : 796.44
> Features        : swp half thumb fastmult edsp
> CPU implementer : 0x56
> CPU architecture: 5TE
> CPU variant     : 0x2
> CPU part        : 0x131
> CPU revision    : 1
> 
> The kernel driver allocates some page aligned memory which is memory
> mapped by the application.
> The driver put some data in the memory area and informs the application.
> The application processes the data, changes it and informs the kernel. The
> kernel then processes the data further and remove it from the buffer.
> 
> We see that the driver doesn't always see the changes of the application to
> the data.
> We see that the changes by the application near the initial kernel data are
> not seen but if data is added on other places in the buffer which where not
> accessed earlier by the kernel, the kernel sees the changes.
> It seems that this area which is not updated in the kernel is the size of a
> cache line of the L1 cache.
> 
> In the mailing list archive I have found the following thread:
> http://lists.arm.linux.org.uk/lurker/message/20091204.112345.0fbed99c.en.
> html
> I tried the suggested patch and it solves the problem described in the thread
> but not our problem.
> The problem in the thread is communication between two user space
> programs as ours is between user space and kernel space.
> 
> I made a small module and application to reproduce the problem (source
> attached).
> When it goes wrong i see the following output:
> 
> [ 1211.827680] Sending packet with counter 0 to user space Received packet
> from kernel with counter 0 [ 1211.834556] removing packet with counter 0
> 
> [ 1216.841009]Sending packet with counter 1 to user space Received packet
> from kernel with counter 1 [ 1216.852027] Value 5 of packet 1 is not equal.
> [ 1216.857752] Value 6 of packet 1 is not equal.
> [ 1216.863119] Value 7 of packet 1 is not equal.
> [ 1216.867478] Value 8 of packet 1 is not equal.
> [ 1216.871838] Value 9 of packet 1 is not equal.
> [ 1216.876190] Value 10 of packet 1 is not equal.
> 
> [ 1221.854343] Sending packet with counter 2 to user space Received packet
> from kernel with counter 3 [ 1226.873237] removing packet with counter 3 [
> 1226.878701] Value 5 of packet 3 is not equal.
> [ 1226.884422] Value 6 of packet 3 is not equal.
> [ 1226.889796] Value 7 of packet 3 is not equal.
> [ 1226.894149] Value 8 of packet 3 is not equal.
> ...
> 
> The output shows that a part of the data written in user space is not read
> back successful in kernel space.
> 
> Regards,
> Bart