Memory inconsistency when mmap()ping

[Christoph M.]

Hello list,

I'm experiencing a strange problem when mmap()ping memory
to userspace on ARM-based Linux System.

What I'd like to achieve is to have a buffer which acts like a
shared memory between my kernel driver and an userspace application.
The driver allocates the buffer and waits for external events
(e.g. timer callbacks). When such an event occurs the driver reads a
value ("event counter") from the buffer, increments it and writes it back
to the buffer. Further the driver releases a blocking call of an
userspace application, which wants to get notified of new events.
The userspace application mmap(2)s the buffer and then starts a loop.
Within this loop it calls poll(2), which blocks until a new event occurs
and reads out the event counter in the buffer.
Of course this is a simplified description, but should be sufficient to
get the idea.

To test the driver, I've further added the ability to get the current
event counter using read(2), which reads out the current value from
the buffer.
My test procedure does the following:
 1) mmap() the buffer
 2) poll() new event
 3) read() the current event counter ("cnt_read")
 4) read the current event counter from the buffer ("cnt_mem")
 5) check if cnt_read == cnt_mem
 6) goto 2)

My problem is, that the check fails almost always.
The cnt_read value increments as expected, but cnt_mem does not.
Instead I see things like this:
 cnt_read (307) != cnt_mem (291)
 cnt_read (308) != cnt_mem (291)
 cnt_read (309) != cnt_mem (291)
 cnt_read (310) != cnt_mem (291)
 cnt_read (311) != cnt_mem (310)
 cnt_read (312) != cnt_mem (311)
 cnt_read (313) != cnt_mem (311)

This means, that a value written to the shared memory can be read
back from the kernel, but userspace reads old values.

Now to the technical details:
 * The memory is allocated using kzalloc(PAGE_SIZE, GFP_KERNEL);
 * The event counter is a u32 value which is stored at offset 0 of
   the buffer.
 * When accessing (read/write) the buffer from kernelspace a call
   to mb() is done afterwards.
 * My mmap() code does this:
     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 * I'm testing with an event intervall of 500 ms. The requirements for
   the driver specify an event intervall of 10 ms.
 * The code works on my x86 development machine.
 * The test machines are a Freescale i.MX53 (Cortex-A8) evaluation board
    and a Freescale i.MX6 (Cortex-A9) evaluation board.
 * Both test machines have a VIPT cache.
 * The i.MX53 runs a 2.6.35 kernel with tons of patches from Freescale's BSP.
 * The i.MX6 runs a 3.2.0 kernel.

If it helps I can provide the source code (kernel driver has ~400 LOC).

I would be very thankful if someone could explain the effect I'm observing.
Of course a pointer to a solution is also welcome. :)
Thanks in advance for any tips/ideas!

BR
Christoph