Looking for pointers for how to determine source of issue

[Nicolas Pitre]

On Thu, 5 May 2011, B.J. Buchalter wrote:

> If I monitor CPU usage on the system using top or atop, I find that 
> the cpu load varies over time (with a periodicity of 15-30 seconds -- 
> it is not really consistent). The CPU load attributed to the program 
> goes from < 1% to over 50% for a short period of time. It is hard to 
> understand why this would happen, as the work being done by the 
> program is consistent (it does not vary over time).

Your program is probably the victim of some IRQ or some other 
non-process context running in the kernel for which the CPU usage is 
charged to whatever process that happens to be current at that time.  
Sometimes such periodicity might be attributed to kernel threads such as 
kswapd that wakes up once in a while to do some cleaning work.

> Does anyone know how to understand this CPU cycling, or investigate it 
> in more detail? On Mac OS X there is a tool called Shark that can do a 
> system trace that shows all thread tenures over time and things like 
> thread context switches, etc. Are there any tools like that available 
> on Arm Linux? Is this something I could use any of the kernel tracing 
> or probing infrastructure to investigate?

You could have a look at http://lttng.org.


Nicolas