Future of git.openwrt.org [Was: Re: Moving git.openwrt.org behind Fastly CDN]

[Paul Spooren]

Hi,

> On Mon, 2025-04-07 at 21:43 +0300, Stijn Tintel wrote:
> Both git-http-backend and gitweb were served by the same fcgiwrap instance, so too many bots crawling gitweb could result in git clone/fetch over HTTP being slow or timing out. Therefore I've enabled a 2nd fcgiwrap instance that is only used for gitweb, and set CPUShares=50 in its systemd service. As a result, the fcgiwrap instance serving normal git+http requests will get prioritized over gitweb requests. When no git+http requests are happening, the gitweb fcgiwrap instance can still use all available CPU.
> 
> It's again another bandaid, and it doesn't solve gitweb from being slow, but at least normal git+http requests are less likely to fail.

Thanks for putting in the work, this seems like a good bandaid for the time being!

> As previously indicated, caching would not be super useful as there are just too many different requests. Making the cache big enough could work, but this would also require us to increase the DO droplet size.
> 
> As for other solutions proposed, maybe it's time to think again about setting up an infra committee with some people with an sysadmin/infrastructure background, and have regular meetings between those people. I've added this on the meeting pad for the next OpenWrt meeting.

This sounds a bit like the embedded OS experts have to become (Git) infrastructure experts and at the same time have to spent more money on DO hardware (or ask for further funding). I’d like to add the idea of handing some money to same minding organizations with a dedicated focus on Git infrastructure. This approach would not only let us focus at what we do best but also strengthens other open source projects.

Being more specific, I talked with codeberg.org <http://codeberg.org/> on how they handle bots. While maintaining blocklists, (form my understanding) they also plan to decapsulate the web interface from the underlying Git access, similar to what you did. As a project, we could migrate our Git to their infrastructure (excl. Issues/PRs/CI hosted on GitHub) and donate a reasonable amount of our donations.

Another thing seems to be running *Anubis* as a proof of work. Not sure if we want to enforce JavaScript https://github.com/TecharoHQ/anubis

Best,
Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20250408/b6d2a11d/attachment.sig>