[GIT PULL] Identity mapping changes for 3.3
Russell King - ARM Linux
linux at arm.linux.org.uk
Wed Dec 7 14:48:26 EST 2011
On Tue, Dec 06, 2011 at 11:25:47PM -0500, Nicolas Pitre wrote:
> Make sure the repo on that machine is nicely packed. Running "git gc"
> (gc as in garbage collect) once in a while is a good thing to do,
> especially that you now have the smart HTTP protocol enabled. That will
> bring the memory usage way down, and serving requests will be much
> faster too. It is safe to put that in a cron job once a week or so,
> even if concurrent requests are being serviced.
Well, I tried an experiment. On my laptop, if I run git fsck, it takes
around about 20 minutes to complete.
On ZenIV, I started this, this morning:
$ GIT_DIR=linux-2.6-arm.git git fsck
and it's now (this evening) some 10 hours after, its still going. This
is the exact same repository (as it's an rsync'd copy of the git objects
and packs which are on the laptop.)
$ ps aux | grep git
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
rmk 26731 1.4 1.0 382716 22036 pts/71 D+ 08:37 9:13 git fsck
apache 29810 1.0 10.0 793288 206564 ? Sl 09:24 6:09 git pack-objects --revs --all --stdout --progress --delta-base-offset
apache 30923 1.0 10.0 790560 206260 ? Sl 09:39 5:53 git pack-objects --revs --all --stdout --progress --delta-base-offset
So, it's consumed 9h of CPU time, the git pack-objects have taken some
5-6 hours...
$ vmstat 10
procs ------------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 4 2502200 45568 8252 125912 50 16 37 62 1 1 1 1 91 7 0
0 4 2502068 46056 8176 126016 44 0 23858 9 2231 1216 1 4 1 94 0
0 4 2501724 45100 8188 126764 76 0 22160 6 2269 1196 1 4 4 91 0
$ iostat 10
...
avg-cpu: %user %nice %system %iowait %steal %idle
0.88 0.00 4.10 91.71 0.00 3.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdf 1.20 10.40 4.80 104 48
sde 0.70 16.00 0.00 160 0
sdd 16.00 432.00 90.40 4320 904
sdg 126.00 12614.40 0.00 126144 0
md0 3.90 26.40 4.80 264 48
sda 139.90 12292.80 0.00 122928 0
sdb 136.10 12393.60 0.00 123936 0
sdc 113.70 12238.40 0.00 122384 0
md1 628.20 49643.20 0.00 496432 0
The repository resides on md1 (sdg, sda, sdb, sdc), the swap partially
on sdd and partially on md0 (sdf and sde).
If I kill all git processes, the disks fall back to about 2MB/s over 10
seconds of analysis.
As you can see, git fsck seems to be pulling data at around 50MB/s,
presumably for 9 hours - this is rediculous because there's only 500MB
of git data for it to read!
What are the top six users of system memory?
rmk 26731 1.4 1.0 380600 21348 pts/71 D+ 08:37 9:17 git fsck
named 30776 0.4 1.4 130804 29212 ? Ssl Oct08 386:55 /usr/sbin/named-sdb
root 2475 0.6 2.0 58940 41376 ? S 10:40 3:38 spamd child
exim 2148 0.0 4.4 160228 91100 ? Ssl Jun24 183:50 clamd.exim
apache 29810 1.0 10.2 794312 210816 ? Sl 09:24 6:27 git pack-objects
apache 30923 1.0 10.3 794656 213536 ? Sl 09:39 6:10 git pack-objects
and lower down there's the bunch of apache httpd processes.
What this is saying to me is that git can't run sensibly on a dual-core
P4, 3GHz machine with 2G of RAM and 4G swap, with a disk IO subsystem
capable of about 50MB/s - basically, git is driving ZenIV into the ground
(and I believe git was also responsible for ZenIV having a load average
hitting a few hundred several months ago which resulted in us having to
have it rebooted.)
Last bit of information - from /proc/29810/maps:
0012b000-00148000 r-xp 00000000 08:33 179255 /lib/ld-2.13.so
00148000-00149000 r--p 0001c000 08:33 179255 /lib/ld-2.13.so
00149000-0014a000 rw-p 0001d000 08:33 179255 /lib/ld-2.13.so
0014c000-002cf000 r-xp 00000000 08:33 179409 /lib/libc-2.13.so
002cf000-002d0000 ---p 00183000 08:33 179409 /lib/libc-2.13.so
002d0000-002d2000 r--p 00183000 08:33 179409 /lib/libc-2.13.so
002d2000-002d3000 rw-p 00185000 08:33 179409 /lib/libc-2.13.so
002d3000-002d6000 rw-p 00000000 00:00 0
00304000-0031b000 r-xp 00000000 08:33 179711 /lib/libpthread-2.13.so
0031b000-0031c000 r--p 00016000 08:33 179711 /lib/libpthread-2.13.so
0031c000-0031d000 rw-p 00017000 08:33 179711 /lib/libpthread-2.13.so
0031d000-0031f000 rw-p 00000000 00:00 0
00328000-0033c000 r-xp 00000000 08:33 179755 /lib/libz.so.1.2.5
0033c000-0033d000 rw-p 00013000 08:33 179755 /lib/libz.so.1.2.5
006da000-006db000 r-xp 00000000 00:00 0 [vdso]
08047000-08171000 r-xp 00000000 08:32 600804 /usr/libexec/git-core/git
08171000-08176000 rw-p 0012a000 08:32 600804 /usr/libexec/git-core/git
08176000-081be000 rw-p 00000000 00:00 0
0a06a000-0f1b0000 rw-p 00000000 00:00 0 [heap]
80a74000-90836000 rw-p 00000000 00:00 0
986bc000-9a6bc000 r--p 00000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
9a6bc000-9c6bc000 r--p 0b000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
9c6bc000-9e6bd000 rw-p 00000000 00:00 0
a0330000-a0331000 ---p 00000000 00:00 0
a0331000-a0d31000 rw-p 00000000 00:00 0
a0d31000-a0d32000 ---p 00000000 00:00 0
a0d32000-a1732000 rw-p 00000000 00:00 0
a1e72000-a3e72000 r--p 08000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a3e72000-a5e72000 r--p 12000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a5e72000-a7e72000 r--p 02000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a8732000-a9e72000 r--p 1a000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a9e72000-abe72000 r--p 05000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
abe72000-ac6fd000 rw-p 00000000 00:00 0
ac6fd000-ae6fd000 r--p 06000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
ae6fd000-af3fb000 rw-p 00000000 00:00 0
af946000-af947000 rw-p 00000000 00:00 0
afb00000-afbb3000 rw-p 00000000 00:00 0
afbb3000-afc00000 ---p 00000000 00:00 0
afcc1000-b1652000 rw-p 00000000 00:00 0
b1800000-b1900000 rw-p 00000000 00:00 0
b1ae8000-b1af0000 r--p 00000000 09:01 88555537 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-60ee04b368ada02ba788d17a537c71bad7471a7a.idx
b1af0000-b1af3000 r--p 00000000 09:01 106318826 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-87f106a487c423809976c6868ec7f1e9fc12e676.idx
b1af3000-b1af6000 r--p 00000000 09:01 88555528 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1a10fd87008d0e06d2ce83efff9bf4ba7abb12e8.idx
b1af6000-b1b03000 r--p 00000000 09:01 88555566 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-6ed1c32d2ec8e0cf20c9ef7d8b813e1f63f67203.idx
b1d00000-b1de8000 rw-p 00000000 00:00 0
b1de8000-b1e00000 ---p 00000000 00:00 0
b1e00000-b1e51000 rw-p 00000000 00:00 0
b1e51000-b1f00000 ---p 00000000 00:00 0
b1f00000-b1ffe000 rw-p 00000000 00:00 0
b1ffe000-b2000000 ---p 00000000 00:00 0
b2000000-b20e5000 rw-p 00000000 00:00 0
b20e5000-b2100000 ---p 00000000 00:00 0
b2100000-b21fc000 rw-p 00000000 00:00 0
b21fc000-b2200000 ---p 00000000 00:00 0
b2200000-b22fd000 rw-p 00000000 00:00 0
b22fd000-b2300000 ---p 00000000 00:00 0
b2300000-b23d7000 rw-p 00000000 00:00 0
b23d7000-b2400000 ---p 00000000 00:00 0
b2400000-b24fd000 rw-p 00000000 00:00 0
b24fd000-b2500000 ---p 00000000 00:00 0
b2500000-b25ea000 rw-p 00000000 00:00 0
b25ea000-b2600000 ---p 00000000 00:00 0
b2600000-b26e4000 rw-p 00000000 00:00 0
b26e4000-b2700000 ---p 00000000 00:00 0
b2700000-b27ee000 rw-p 00000000 00:00 0
b27ee000-b2800000 ---p 00000000 00:00 0
b2800000-b28ec000 rw-p 00000000 00:00 0
b28ec000-b2900000 ---p 00000000 00:00 0
b2900000-b29f3000 rw-p 00000000 00:00 0
b29f3000-b2a00000 ---p 00000000 00:00 0
b2b00000-b2cfe000 rw-p 00000000 00:00 0
b2cfe000-b2d00000 ---p 00000000 00:00 0
b2d00000-b2dfd000 rw-p 00000000 00:00 0
b2dfd000-b2e00000 ---p 00000000 00:00 0
b2f00000-b2ff4000 rw-p 00000000 00:00 0
b2ff4000-b3000000 ---p 00000000 00:00 0
b3100000-b31fe000 rw-p 00000000 00:00 0
b31fe000-b3200000 ---p 00000000 00:00 0
b3300000-b33f1000 rw-p 00000000 00:00 0
b33f1000-b3400000 ---p 00000000 00:00 0
b3500000-b35f6000 rw-p 00000000 00:00 0
b35f6000-b3600000 ---p 00000000 00:00 0
b3600000-b36f3000 rw-p 00000000 00:00 0
b36f3000-b3700000 ---p 00000000 00:00 0
b3700000-b37f9000 rw-p 00000000 00:00 0
b37f9000-b3800000 ---p 00000000 00:00 0
b3900000-b39f3000 rw-p 00000000 00:00 0
b39f3000-b3a00000 ---p 00000000 00:00 0
b3b03000-b76ab000 r--p 00000000 09:01 106316443 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.idx
b76ab000-b76b1000 r--p 00000000 09:01 106316389 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-742507456b81724d4b1b7c62a9c7b1a08dea788c.idx
b76b1000-b76b3000 r--p 00000000 09:01 88555543 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-01d8e6a80f43c483b170f252a63549c3f3177435.idx
b76b3000-b76b5000 r--p 00000000 09:01 106316378 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-22db475e3babd502f5462dbd53b0083f63054da8.idx
b76b5000-b76b7000 r--p 00000000 09:01 106319097 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-81e9156c16956b1ca13d0786f26c57da8a8c4d00.idx
b76b7000-b76b9000 r--p 00000000 09:01 106316404 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-02bd0e7dcc0d4f9ba1faef6cb90d3080451223ab.idx
b76b9000-b76be000 r--p 00000000 09:01 106320451 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-64a9ef29c0dd71c0ada102c21a6d3297715b5ec0.idx
b76be000-b76c0000 r--p 00000000 09:01 88555532 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-cf138203ddca32052979b5eadc8b03fa8d9f6313.idx
b76c0000-b76c2000 r--p 00000000 09:01 88555522 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-08c44c7f188425373f8ecb95e4bb342ee40d340e.idx
b76c2000-b76cb000 r--p 00000000 09:01 88555534 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1fd3ce5f1145636ad725dec83f5ef91b80ec1cf3.idx
b76cb000-b76cf000 r--p 00000000 09:01 88555545 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-0e5768a969e7c4ef4f3406a129b1271971ee5342.idx
b76cf000-b76d4000 r--p 00000000 09:01 88555547 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-92592ad40e58a75d3d1b30a0a46e7ad63817637b.idx
b76d4000-b76d6000 r--p 00000000 09:01 88555549 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-5b26c4eb6fed3ce5b26e921aee548a7f3259a11e.idx
b76d6000-b76d9000 r--p 00000000 09:01 88555551 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4a9b526f37540b18dd2ff4e97ec94fcb49c0aeb6.idx
b76d9000-b76dd000 r--p 00000000 09:01 88555553 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1a7d19bf1f4aa63bb54b0f7e77f5df8889099e3f.idx
b76dd000-b76e0000 r--p 00000000 09:01 88555558 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-7da553217dc96aa89cdcf50fe45615143aa01089.idx
b76e0000-b76e6000 r--p 00000000 09:01 88555555 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-e795f1594ea70cca88a97ad23d2d62da06941038.idx
b76e6000-b76ea000 r--p 00000000 09:01 88555560 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4c79da67fb0435e3a4a7440a714fb9d617995387.idx
b76e6000-b76ea000 r--p 00000000 09:01 88555560 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4c79da67fb0435e3a4a7440a714fb9d617995387.idx
b76ea000-b76ef000 r--p 00000000 09:01 88555563 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-63e7f343a0aa5f8f6841c81211db00ce95adbe76.idx
b76ef000-b76f1000 rw-p 00000000 00:00 0
b76f5000-b76fa000 r--p 00000000 09:01 88555565 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-b37609403c59702e384e42a9df4bc16e2b3c1072.idx
b76fa000-b76fc000 r--p 00000000 09:01 88555542 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-23f0d9e30ada6f829f6743a395bcb6f229c1453b.idx
b76fc000-b7700000 r--p 00000000 09:01 88555571 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-d073eba6bd4143e405ec9db4172d4406fa08ace8.idx
b7700000-b7702000 r--p 00000000 09:01 88555574 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-b2481dc10cc72226ff396e864e4a6d7e2c26137d.idx
b7702000-b7704000 r--p 00000000 09:01 88555577 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-866787d1e38a3c7c9ac0ab7057935bf5970bf319.idx
b7704000-b7705000 rw-p 00000000 00:00 0
bf8dc000-bf8fd000 rw-p 00000000 00:00 0 [stack]
It's worth noting that Linus tree currently has 19 pack files, my tree
has an additional 9 on that.
More information about the linux-arm-kernel
mailing list