Forum host upgrade - proposal

[Ted Hess]

Johnathan - Not sure what is your point here. Do you want to do some 
work on the forum performance? All help is appreciated. I'm not sure the 
areas you suggest would make a significant reduction in use given the 
shear size and popularity of the site.

With-respect-to (wrt) Crawlers - A typical bad day breaks down like 
this:
     15K Logged in users
     110K Anonymous users
     47K Crawlers

Biggest offenders are the major indexers - Bing, Google, Yandex &  
Baidu. In general it is not a good idea to block these. However, we do 
slow them down by rate-limiting based on user-agent. People masquerade 
as other user-agents to try and bypass this. We also block a number of 
known offenders entirely. It's a moving target.

Wrt email - This one is difficult. Especially, since we allow users to 
interact with the forum via email. Folks (like myself and others) get 
summaries mailed by option - you can turn yours off. Others prefer to 
receive notices of new posts on their favorite topics. I haven't really 
broken down the details of how many of what type of emails are being 
sent. I do remember a while back I looked at the number of users getting 
summaries and it was a reasonable number given the size of the forum.


On 8/10/2024 3:22:43 PM, "Jonathan lancett" <lancett01 at googlemail.com> 
wrote:

>25% attributed to crawlers is a lot. Can we do something about that?
>Is there a way of cutting down on the mail messages?
>
>
>On 10/08/2024 16:53, Ted Hess wrote:
>>Hi all -
>>
>>Those of you who visit the forum have probably noticed its, sometimes extreme, sluggishness over the last several months. Since April, I have worked on and applied a number of attempts at tuning, cleanup and storage re-arrangement to try and alleviate these issues. As best as I can determine, we have have just simply exceeded the capacity of the current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo. The difference in cost is a "Regular Intel" vs "Premium Intel" processors. The Premium (latest Xeon) option will provide enhanced NVMe SSDs which would improve DB performance (a bit). I think it would be worth $112/mo for maximal impact on the responsiveness of the forum. AMD vs Intel - cost is the same. AMD may be slightly higher GHz.
>>
>>Are there any objections / reasons why we shouldn't go ahead and do the upgrade. There are a couple of methods available however, the down-time will probably be 1-3hrs. if no problems. This isn't the first system upgrade we have done, so I expect no issues.
>>
>>If any of you wish to see some of the forum performance stats, I can post another message with numbers and personal observations. A quick overview of what our daily forum activity looks like:
>>
>>* Serving 120-160K page views. (about 25% attributed to crawlers - a different discussion)
>>* 6-8K mail messages generated
>>* ~1100 User visits (signed-in)
>>* ~200 engaged users (liked or posted)
>>* ~500 new posts
>>
>>Footnote on bandwidth usage... (DO charges bandwidth on aggregate Droplet use)
>>Proposed Droplet upgrade will add 3TB to our total.
>>I don't have per-Droplet stats, but last month we transmitted ~29TB of our 25TB allowance. A $40 overage cost.
>>
>>
>>_______________________________________________
>>openwrt-adm mailing list
>>openwrt-adm at lists.openwrt.org
>>https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>
>
>
>_______________________________________________
>openwrt-adm mailing list
>openwrt-adm at lists.openwrt.org
>https://lists.openwrt.org/mailman/listinfo/openwrt-adm