Future of git.openwrt.org [Was: Re: Moving git.openwrt.org behind Fastly CDN]

[David Woodhouse]

On Mon, 2025-04-07 at 21:43 +0300, Stijn Tintel wrote:
> On 6/12/2024 7:59 a.m., Petr Štetiar wrote:
> > As a quick fix, I've done following in the past days:
> > 
> >    - disabled tarballs for everyone with 403
> >    - enabled IP based rate limits on everyone
> > 
> >      * heavy projects like luci.git, packages.git and openwrt.git
> > 
> >        - after 5r/m additional requests are delayed up to 15r/m, then 429 sorry
> > 
> >      * other requests 15r/m, delayed after 8-th request, up to 30r/m, then 429 sorry
> 
> This morning around 7:00 UTC the git server got overloaded again and as 
> a result a lot of buildbot jobs started failing. The logs show a lot of 
> 429, so the rate limiting is definitely doing something, but it appears 
> it's not enough. I've implemented some additional changes to avoid the 
> buildbot jobs from failing again when the server is under high load.
> 
> Both git-http-backend and gitweb were served by the same fcgiwrap 
> instance, so too many bots crawling gitweb could result in git 
> clone/fetch over HTTP being slow or timing out. Therefore I've enabled a 
> 2nd fcgiwrap instance that is only used for gitweb, and set CPUShares=50 
> in its systemd service. As a result, the fcgiwrap instance serving 
> normal git+http requests will get prioritized over gitweb requests. When 
> no git+http requests are happening, the gitweb fcgiwrap instance can 
> still use all available CPU.
> 
> It's again another bandaid, and it doesn't solve gitweb from being slow, 
> but at least normal git+http requests are less likely to fail.

On git.infradead.org I've disabled the ;a=history action in gitweb. I
only did it this morning but it seems promising so far...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5069 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20250407/2808e95f/attachment.p7s>