Wiki performance and apparent DDoS attack

[Baptiste Jonglez]

Hi Ted,

On 21-02-25, Ted Hess wrote:
> Over the last month or so, I've been monitoring wiki web access and
> performance. Myself, and a couple others,  have a pretty good handle on
> site-scrapers, bots and LLM scavengers. We have instituted a number of
> rate-limiting, IP blocking and UA blocking rules. This action had greatly
> reduced the load on the wiki.
> 
> Recently, the site has been overloaded again with certain ToH queries which
> have long running times and were timing out. Failures were logging Nginx 499
> (proxy or client timeout) errors for the PHP-FPM processes. Perhaps these
> queries were lost in the noise created by other bots. It was the sheer
> number of 499 errors that got my attention. Analyses on our logs showed
> >1,.5M timeouts/day on a single page with varying query strings. The
> accesses looked like legitimate browser queries from Windows platforms - I
> believe this to be totally bogus. Attempts at rate-limiting access to this
> page based on IP address was not very successful. It turns out that over 1M
> of these queries are from unique IP address within >8K subnets located
> around the world.
> 
> This is definitely an application level DDoS attack. As of today, I have
> blocked all access and turned off logging for queries to the ToH Performance
> page. Enjoy the new found responsiveness of the wiki ;)

Thanks a lot for the action and well done for getting to the bottom of
this, this is good detective work!

As you know, I also occasionally monitor wiki performance and block/limit
new abusers with your rules.  I tried to track down this wave a few days
ago, but couldn't understand where the load was coming from.

Long term, we definitely need to move away from this expensive ToH page system.

Regards,
Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20250222/6ee751da/attachment.sig>