Wiki performance and apparent DDoS attack

[Ted Hess]

Hi all -

Over the last month or so, I've been monitoring wiki web access and 
performance. Myself, and a couple others,  have a pretty good handle on 
site-scrapers, bots and LLM scavengers. We have instituted a number of 
rate-limiting, IP blocking and UA blocking rules. This action had 
greatly reduced the load on the wiki.

Recently, the site has been overloaded again with certain ToH queries 
which have long running times and were timing out. Failures were logging 
Nginx 499 (proxy or client timeout) errors for the PHP-FPM processes. 
Perhaps these queries were lost in the noise created by other bots. It 
was the sheer number of 499 errors that got my attention. Analyses on 
our logs showed >1,.5M timeouts/day on a single page with varying query 
strings. The accesses looked like legitimate browser queries from 
Windows platforms - I believe this to be totally bogus. Attempts at 
rate-limiting access to this page based on IP address was not very 
successful. It turns out that over 1M of these queries are from unique 
IP address within >8K subnets located around the world.

This is definitely an application level DDoS attack. As of today, I have 
blocked all access and turned off logging for queries to the ToH 
Performance page. Enjoy the new found responsiveness of the wiki ;)

/ted