[BUG] 2.6.37-rc3 massive interactivity regression on ARM

[Russell King - ARM Linux]

On Sun, Dec 05, 2010 at 01:17:02PM +0000, Russell King - ARM Linux wrote:
> On Sun, Dec 05, 2010 at 01:32:37PM +0100, Mikael Pettersson wrote:
> > Mikael Pettersson writes:
> >  > The scenario is that I do a remote login to an ARM build server,
> >  > use screen to start a sub-shell, in that shell start a largish
> >  > compile job, detach from that screen, and from the original login
> >  > shell I occasionally monitor the compile job with top or ps or
> >  > by attaching to the screen.
> >  > 
> >  > With kernels 2.6.37-rc2 and -rc3 this causes the machine to become
> >  > very sluggish: top takes forever to start, once started it shows no
> >  > activity from the compile job (it's as if it's sleeping on a lock),
> >  > and ps also takes forever and shows no activity from the compile job.
> >  > 
> >  > Rebooting into 2.6.36 eliminates these issues.
> >  > 
> >  > I do pretty much the same thing (remote login -> screen -> compile job)
> >  > on other archs, but so far I've only seen the 2.6.37-rc misbehaviour
> >  > on ARM EABI, specifically on an IOP n2100. (I have access to other ARM
> >  > sub-archs, but haven't had time to test 2.6.37-rc on them yet.)
> >  > 
> >  > Has anyone else seen this? Any ideas about the cause?
> > 
> > (Re-followup since I just realised my previous followups were to Rafael's
> > regressions mailbot rather than the original thread.)
> > 
> > > The bug is still present in 2.6.37-rc4.  I'm currently trying to bisect it.
> > 
> > git bisect identified
> > 
> > [305e6835e05513406fa12820e40e4a8ecb63743c] sched: Do not account irq time to current task
> > 
> > as the cause of this regression.  Reverting it from 2.6.37-rc4 (requires some
> > hackery due to subsequent changes in the same area) restores sane behaviour.
> > 
> > The original patch submission talks about irq-heavy scenarios.  My case is the
> > exact opposite: UP, !PREEMPT, NO_HZ, very low irq rate, essentially 100% CPU
> > bound in userspace but expected to schedule quickly when needed (e.g. running
> > top or ps or just hitting CR in one shell while another runs a compile job).
> > 
> > I've reproduced the misbehaviour with 2.6.37-rc4 on ARM/mach-iop32x and
> > ARM/mach-ixp4xx, but ARM/mach-kirkwood does not misbehave, and other archs
> > (x86 SMP, SPARC64 UP and SMP, PowerPC32 UP, Alpha UP) also do not misbehave.
> > 
> > So it looks like an ARM-only issue, possibly depending on platform specifics.
> > 
> > One difference I noticed between my Kirkwood machine and my ixp4xx and iop32x
> > machines is that even though all have CONFIG_NO_HZ=y, the timer irq rate is
> > much higher on Kirkwood, even when the machine is idle.
> 
> The above patch you point out is fundamentally broken.
> 
> +               rq->clock = sched_clock_cpu(cpu);
> +               irq_time = irq_time_cpu(cpu);
> +               if (rq->clock - irq_time > rq->clock_task)
> +                       rq->clock_task = rq->clock - irq_time;
> 
> This means that we will only update rq->clock_task if it is smaller than
> rq->clock.  So, eventually over time, rq->clock_task becomes the maximum
> value that rq->clock can ever be.  Or in other words, the maximum value
> of sched_clock_cpu().
> 
> Once that has been reached, although rq->clock will wrap back to zero,
> rq->clock_task will not, and so (I think) task execution time accounting
> effectively stops dead.
> 
> I guess this hasn't been noticed on x86 as they have a 64-bit sched_clock,
> and so need to wait a long time for this to be noticed.  However, on ARM
> where we tend to have 32-bit counters feeding sched_clock(), this value
> will wrap far sooner.

I'm not so sure about this - certainly that if() statement looks very
suspicious above.  As irq_time_cpu() will always be zero, can you try
removing the conditional?

In any case, sched_clock_cpu() should be resilient against sched_clock()
wrapping.  However, your comments about it being iop32x and ixp4xx
(both of which are 32-bit-counter-to-ns based implementations) and
kirkwood being a 32-bit-extended-to-63-bit-counter-to-ns implementation
does make me wonder...