Forum host upgrade - proposal

[Etienne Champetier]

Hi Tess,

First thanks for all the work !

Le sam. 10 août 2024 à 11:54, Ted Hess <thess at kitschensync.net> a écrit :
>
> Hi all -
>
> Those of you who visit the forum have probably noticed its, sometimes
> extreme, sluggishness over the last several months. Since April, I have
> worked on and applied a number of attempts at tuning, cleanup and
> storage re-arrangement to try and alleviate these issues. As best as I
> can determine, we have have just simply exceeded the capacity of the
> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> The difference in cost is a "Regular Intel" vs "Premium Intel"
> processors. The Premium (latest Xeon) option will provide enhanced NVMe
> SSDs which would improve DB performance (a bit). I think it would be
> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> vs Intel - cost is the same. AMD may be slightly higher GHz.

How up to date is the wiki infrastructure page
(https://openwrt.org/infrastructure#servers) ?
Is Digital Ocean still offering some credits ?

At 48$ on DO website I see we are talking about 'basic droplets', i.e.
shared CPUs,
so the first question is how much cpu steals time do you see ?
If we see high cpu steal time, it might be better to go dedicated
(general purpose) with the same size rather than bigger.

For the SSD performance (high IO wait ?)
I have no idea if you are using the local storage for the DB and if
that would help on DO, but sharing just in case.
If TRIM is not used/can't be used (RAID card/how DO does the
virtualisation/...),
I've seen SSD crawl to 1/10 of their performance after some time.
A trick is to fill your partition free space with 0s multiple times,
if the SSD firmware is smart enough it'll do a trim for the blocks
where you put 0s.
(dd if=/dev/zero of=/bigfile status=progress; rm -f /bigfile)

To throw more power at the problem, another solution is renting bare
metal servers.
For example at OVH, If you look at a SYS-6-SSD-128, for $64.99/month you have
- AMD Epyc 7351p - 16c/32t - 2.4 GHz/2.9 GHz (same generation as DO
non premium / x8 vCPUs)
- 128GB DDR4 ECC 2400MHz (x16 mem)
- 2× 500GB SSD NVMe Soft RAID (??)
- 500Mbps unmetered internet connection

The big catches I can think of:
- you pay 1 month installation fee so you can't easily change sizes
- storage is not flexible at all
- no easy full system backup / snapshots
(I'm a happy OVH customer for years / not affiliated)

Best
Etienne

> Are there any objections / reasons why we shouldn't go ahead and do the
> upgrade. There are a couple of methods available however, the down-time
> will probably be 1-3hrs. if no problems. This isn't the first system
> upgrade we have done, so I expect no issues.
>
> If any of you wish to see some of the forum performance stats, I can
> post another message with numbers and personal observations. A quick
> overview of what our daily forum activity looks like:
>
> * Serving 120-160K page views. (about 25% attributed to crawlers - a
> different discussion)
> * 6-8K mail messages generated
> * ~1100 User visits (signed-in)
> * ~200 engaged users (liked or posted)
> * ~500 new posts
>
> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> Droplet use)
> Proposed Droplet upgrade will add 3TB to our total.
> I don't have per-Droplet stats, but last month we transmitted ~29TB of
> our 25TB allowance. A $40 overage cost.