Future of git.openwrt.org [Was: Re: Moving git.openwrt.org behind Fastly CDN]

[Stijn Tintel]

On 6/12/2024 7:59 a.m., Petr Štetiar wrote:
> As a quick fix, I've done following in the past days:
>
>   - disabled tarballs for everyone with 403
>   - enabled IP based rate limits on everyone
>
>     * heavy projects like luci.git, packages.git and openwrt.git
>
>       - after 5r/m additional requests are delayed up to 15r/m, then 429 sorry
>
>     * other requests 15r/m, delayed after 8-th request, up to 30r/m, then 429 sorry

This morning around 7:00 UTC the git server got overloaded again and as 
a result a lot of buildbot jobs started failing. The logs show a lot of 
429, so the rate limiting is definitely doing something, but it appears 
it's not enough. I've implemented some additional changes to avoid the 
buildbot jobs from failing again when the server is under high load.

Both git-http-backend and gitweb were served by the same fcgiwrap 
instance, so too many bots crawling gitweb could result in git 
clone/fetch over HTTP being slow or timing out. Therefore I've enabled a 
2nd fcgiwrap instance that is only used for gitweb, and set CPUShares=50 
in its systemd service. As a result, the fcgiwrap instance serving 
normal git+http requests will get prioritized over gitweb requests. When 
no git+http requests are happening, the gitweb fcgiwrap instance can 
still use all available CPU.

It's again another bandaid, and it doesn't solve gitweb from being slow, 
but at least normal git+http requests are less likely to fail.

As previously indicated, caching would not be super useful as there are 
just too many different requests. Making the cache big enough could 
work, but this would also require us to increase the DO droplet size.

As for other solutions proposed, maybe it's time to think again about 
setting up an infra committee with some people with an 
sysadmin/infrastructure background, and have regular meetings between 
those people. I've added this on the meeting pad for the next OpenWrt 
meeting.

Best,
Stijn