Future of git.openwrt.org [Was: Re: Moving git.openwrt.org behind Fastly CDN]

[Ted Hess]

Hey folks,

I have Anubis running on our main wiki site since yesterday -- so far, 
so good. I still have some rules tuning to do regarding how we treat 
legitimate web crawlers vs bot scrapers and AI LLM data gatherers. We 
can still apply rate-limiting rules to agents allowed in without 
challenges as well as completely deny access to agents, ip addresses, 
URIs, etc. we don't accept.

Do you think it would help to front-end our git repo with Anubis so one 
does not need to play whack-a-mole as we have done on the wiki up to 
now? I was also looking at another similar tool called go-away, but it 
looks harder to deploy and is really still under development.

/ted

On 5/20/2025 9:46:46 AM, "Petr Štetiar" <ynezz at true.cz> wrote:

>David Woodhouse <dwmw2 at infradead.org> [2025-04-07 20:42:45]:
>
>Hi,
>
>>  On Mon, 2025-04-07 at 21:43 +0300, Stijn Tintel wrote:
>>  >
>>  > It's again another bandaid, and it doesn't solve gitweb from being slow,
>>  > but at least normal git+http requests are less likely to fail.
>>
>>  On git.infradead.org I've disabled the ;a=history action in gitweb. I
>>  only did it this morning but it seems promising so far...
>
>another wave of AI scrappers hit the box, I've added some additional band
>aids and disabled the history feature as well. Load is back to normal.
>
>Cheers,
>
>Petr
>
>_______________________________________________
>openwrt-adm mailing list
>openwrt-adm at lists.openwrt.org
>https://lists.openwrt.org/mailman/listinfo/openwrt-adm