Looking for pointers for how to determine source of issue

[B.J. Buchalter]

On May 5, 2011, at 3:18 PM, Nicolas Pitre wrote:

> On Thu, 5 May 2011, B.J. Buchalter wrote:
> 
>> If I monitor CPU usage on the system using top or atop, I find that 
>> the cpu load varies over time (with a periodicity of 15-30 seconds -- 
>> it is not really consistent). The CPU load attributed to the program 
>> goes from < 1% to over 50% for a short period of time. It is hard to 
>> understand why this would happen, as the work being done by the 
>> program is consistent (it does not vary over time).
> 
> Your program is probably the victim of some IRQ or some other 
> non-process context running in the kernel for which the CPU usage is 
> charged to whatever process that happens to be current at that time.  
> Sometimes such periodicity might be attributed to kernel threads such as 
> kswapd that wakes up once in a while to do some cleaning work.

There is no swap on my system.

Even so, I see the kernel level calls charged to my program via the oprofile
run; as I said, it does not appear to change between the "high" cpu load 
case and the "normal" cpu load case. oprofile definitely captures the soft_irq 
handlers and such.

Also, those threads have their own entries in the top process list; wouldn't 
such a wakeup get attributed to the kernel thread in top?

> 
>> Does anyone know how to understand this CPU cycling, or investigate it 
>> in more detail? On Mac OS X there is a tool called Shark that can do a 
>> system trace that shows all thread tenures over time and things like 
>> thread context switches, etc. Are there any tools like that available 
>> on Arm Linux? Is this something I could use any of the kernel tracing 
>> or probing infrastructure to investigate?
> 
> You could have a look at http://lttng.org.

I have been looking at it for the last couple of hours. It might be the 
ticket, but I am concerned that the kernel patches may conflict with the 
patches for the TI816x. I guess the only way to know is to try it.

I was hoping that perhaps someone had actually tried some of this out 
already; LTTng seems pretty complex, and it could be days of work just
to find out that it won't really help....

Are there any other similar tools available?

Best regards,

B.J. Buchalter
Metric Halo 
http://www.mhlabs.com