From thess at kitschensync.net  Sat Aug 10 08:53:45 2024
From: thess at kitschensync.net (Ted Hess)
Date: Sat, 10 Aug 2024 15:53:45 +0000
Subject: Forum host upgrade - proposal
Message-ID: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>

Hi all -

Those of you who visit the forum have probably noticed its, sometimes 
extreme, sluggishness over the last several months. Since April, I have 
worked on and applied a number of attempts at tuning, cleanup and 
storage re-arrangement to try and alleviate these issues. As best as I 
can determine, we have have just simply exceeded the capacity of the 
current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM 
(4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo. 
The difference in cost is a "Regular Intel" vs "Premium Intel" 
processors. The Premium (latest Xeon) option will provide enhanced NVMe 
SSDs which would improve DB performance (a bit). I think it would be 
worth $112/mo for maximal impact on the responsiveness of the forum. AMD 
vs Intel - cost is the same. AMD may be slightly higher GHz.

Are there any objections / reasons why we shouldn't go ahead and do the 
upgrade. There are a couple of methods available however, the down-time 
will probably be 1-3hrs. if no problems. This isn't the first system 
upgrade we have done, so I expect no issues.

If any of you wish to see some of the forum performance stats, I can 
post another message with numbers and personal observations. A quick 
overview of what our daily forum activity looks like:

* Serving 120-160K page views. (about 25% attributed to crawlers - a 
different discussion)
* 6-8K mail messages generated
* ~1100 User visits (signed-in)
* ~200 engaged users (liked or posted)
* ~500 new posts

Footnote on bandwidth usage... (DO charges bandwidth on aggregate 
Droplet use)
Proposed Droplet upgrade will add 3TB to our total.
I don't have per-Droplet stats, but last month we transmitted ~29TB of 
our 25TB allowance. A $40 overage cost.



From lancett01 at googlemail.com  Sat Aug 10 12:22:43 2024
From: lancett01 at googlemail.com (Jonathan lancett)
Date: Sat, 10 Aug 2024 20:22:43 +0100
Subject: Forum host upgrade - proposal
In-Reply-To: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
Message-ID: <26b5b424-ff6f-4f8c-983b-04347faadc85@googlemail.com>

25% attributed to crawlers is a lot. Can we do something about that?
Is there a way of cutting down on the mail messages?


On 10/08/2024 16:53, Ted Hess wrote:
> Hi all -
>
> Those of you who visit the forum have probably noticed its, sometimes 
> extreme, sluggishness over the last several months. Since April, I 
> have worked on and applied a number of attempts at tuning, cleanup and 
> storage re-arrangement to try and alleviate these issues. As best as I 
> can determine, we have have just simply exceeded the capacity of the 
> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM 
> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to 
> $96-112/mo. The difference in cost is a "Regular Intel" vs "Premium 
> Intel" processors. The Premium (latest Xeon) option will provide 
> enhanced NVMe SSDs which would improve DB performance (a bit). I think 
> it would be worth $112/mo for maximal impact on the responsiveness of 
> the forum. AMD vs Intel - cost is the same. AMD may be slightly higher 
> GHz.
>
> Are there any objections / reasons why we shouldn't go ahead and do 
> the upgrade. There are a couple of methods available however, the 
> down-time will probably be 1-3hrs. if no problems. This isn't the 
> first system upgrade we have done, so I expect no issues.
>
> If any of you wish to see some of the forum performance stats, I can 
> post another message with numbers and personal observations. A quick 
> overview of what our daily forum activity looks like:
>
> * Serving 120-160K page views. (about 25% attributed to crawlers - a 
> different discussion)
> * 6-8K mail messages generated
> * ~1100 User visits (signed-in)
> * ~200 engaged users (liked or posted)
> * ~500 new posts
>
> Footnote on bandwidth usage... (DO charges bandwidth on aggregate 
> Droplet use)
> Proposed Droplet upgrade will add 3TB to our total.
> I don't have per-Droplet stats, but last month we transmitted ~29TB of 
> our 25TB allowance. A $40 overage cost.
>
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm




From thess at kitschensync.net  Sat Aug 10 13:19:33 2024
From: thess at kitschensync.net (Ted Hess)
Date: Sat, 10 Aug 2024 20:19:33 +0000
Subject: Forum host upgrade - proposal
In-Reply-To: <26b5b424-ff6f-4f8c-983b-04347faadc85@googlemail.com>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <26b5b424-ff6f-4f8c-983b-04347faadc85@googlemail.com>
Message-ID: <emcf5c5fd8-6054-46bf-8deb-1489730999a7@twonky7>

Johnathan - Not sure what is your point here. Do you want to do some 
work on the forum performance? All help is appreciated. I'm not sure the 
areas you suggest would make a significant reduction in use given the 
shear size and popularity of the site.

With-respect-to (wrt) Crawlers - A typical bad day breaks down like 
this:
     15K Logged in users
     110K Anonymous users
     47K Crawlers

Biggest offenders are the major indexers - Bing, Google, Yandex &  
Baidu. In general it is not a good idea to block these. However, we do 
slow them down by rate-limiting based on user-agent. People masquerade 
as other user-agents to try and bypass this. We also block a number of 
known offenders entirely. It's a moving target.

Wrt email - This one is difficult. Especially, since we allow users to 
interact with the forum via email. Folks (like myself and others) get 
summaries mailed by option - you can turn yours off. Others prefer to 
receive notices of new posts on their favorite topics. I haven't really 
broken down the details of how many of what type of emails are being 
sent. I do remember a while back I looked at the number of users getting 
summaries and it was a reasonable number given the size of the forum.


On 8/10/2024 3:22:43 PM, "Jonathan lancett" <lancett01 at googlemail.com> 
wrote:

>25% attributed to crawlers is a lot. Can we do something about that?
>Is there a way of cutting down on the mail messages?
>
>
>On 10/08/2024 16:53, Ted Hess wrote:
>>Hi all -
>>
>>Those of you who visit the forum have probably noticed its, sometimes extreme, sluggishness over the last several months. Since April, I have worked on and applied a number of attempts at tuning, cleanup and storage re-arrangement to try and alleviate these issues. As best as I can determine, we have have just simply exceeded the capacity of the current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo. The difference in cost is a "Regular Intel" vs "Premium Intel" processors. The Premium (latest Xeon) option will provide enhanced NVMe SSDs which would improve DB performance (a bit). I think it would be worth $112/mo for maximal impact on the responsiveness of the forum. AMD vs Intel - cost is the same. AMD may be slightly higher GHz.
>>
>>Are there any objections / reasons why we shouldn't go ahead and do the upgrade. There are a couple of methods available however, the down-time will probably be 1-3hrs. if no problems. This isn't the first system upgrade we have done, so I expect no issues.
>>
>>If any of you wish to see some of the forum performance stats, I can post another message with numbers and personal observations. A quick overview of what our daily forum activity looks like:
>>
>>* Serving 120-160K page views. (about 25% attributed to crawlers - a different discussion)
>>* 6-8K mail messages generated
>>* ~1100 User visits (signed-in)
>>* ~200 engaged users (liked or posted)
>>* ~500 new posts
>>
>>Footnote on bandwidth usage... (DO charges bandwidth on aggregate Droplet use)
>>Proposed Droplet upgrade will add 3TB to our total.
>>I don't have per-Droplet stats, but last month we transmitted ~29TB of our 25TB allowance. A $40 overage cost.
>>
>>
>>_______________________________________________
>>openwrt-adm mailing list
>>openwrt-adm at lists.openwrt.org
>>https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>
>
>
>_______________________________________________
>openwrt-adm mailing list
>openwrt-adm at lists.openwrt.org
>https://lists.openwrt.org/mailman/listinfo/openwrt-adm



From richb.hanover at gmail.com  Sat Aug 10 18:30:34 2024
From: richb.hanover at gmail.com (Rich Brown)
Date: Sat, 10 Aug 2024 21:30:34 -0400
Subject: Forum host upgrade - proposal
In-Reply-To: <emcf5c5fd8-6054-46bf-8deb-1489730999a7@twonky7>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <26b5b424-ff6f-4f8c-983b-04347faadc85@googlemail.com>
 <emcf5c5fd8-6054-46bf-8deb-1489730999a7@twonky7>
Message-ID: <C4F31C05-034F-4277-BDA4-5BFBC04E446A@gmail.com>

Ted: Thanks for:

a) doing so much heavy lifting on the forum. 
b) thinking about ways to keep the server responsive.

I'm glad you are paying attention.

I submit that it's entirely reasonable to spend an additional $50-$60/month to support this critical piece of our infrastructure. It's one of the most important places that people get information about OpenWrt. If the alternative is to spend our administrative efforts making the server "less useful", I vote for spending the money.

Last time I looked, we're spending something like $250? per month on servers. Increasing this to 300? would not make much of a difference. As long as we have the funding - which I believe we do - from SFC, other resources, or donations (https://openwrt.org/donate) it makes sense to keep the forum server responsive. This is a good use of money.

Thanks again.

Rich

> On Aug 10, 2024, at 4:19 PM, Ted Hess <thess at kitschensync.net> wrote:
> 
> Johnathan - Not sure what is your point here. Do you want to do some work on the forum performance? All help is appreciated. I'm not sure the areas you suggest would make a significant reduction in use given the shear size and popularity of the site.
> 
> With-respect-to (wrt) Crawlers - A typical bad day breaks down like this:
>    15K Logged in users
>    110K Anonymous users
>    47K Crawlers
> 
> Biggest offenders are the major indexers - Bing, Google, Yandex &  Baidu. In general it is not a good idea to block these. However, we do slow them down by rate-limiting based on user-agent. People masquerade as other user-agents to try and bypass this. We also block a number of known offenders entirely. It's a moving target.
> 
> Wrt email - This one is difficult. Especially, since we allow users to interact with the forum via email. Folks (like myself and others) get summaries mailed by option - you can turn yours off. Others prefer to receive notices of new posts on their favorite topics. I haven't really broken down the details of how many of what type of emails are being sent. I do remember a while back I looked at the number of users getting summaries and it was a reasonable number given the size of the forum.
> 
> 
> On 8/10/2024 3:22:43 PM, "Jonathan lancett" <lancett01 at googlemail.com> wrote:
> 
>> 25% attributed to crawlers is a lot. Can we do something about that?
>> Is there a way of cutting down on the mail messages?
>> 
>> 
>> On 10/08/2024 16:53, Ted Hess wrote:
>>> Hi all -
>>> 
>>> Those of you who visit the forum have probably noticed its, sometimes extreme, sluggishness over the last several months. Since April, I have worked on and applied a number of attempts at tuning, cleanup and storage re-arrangement to try and alleviate these issues. As best as I can determine, we have have just simply exceeded the capacity of the current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo. The difference in cost is a "Regular Intel" vs "Premium Intel" processors. The Premium (latest Xeon) option will provide enhanced NVMe SSDs which would improve DB performance (a bit). I think it would be worth $112/mo for maximal impact on the responsiveness of the forum. AMD vs Intel - cost is the same. AMD may be slightly higher GHz.
>>> 
>>> Are there any objections / reasons why we shouldn't go ahead and do the upgrade. There are a couple of methods available however, the down-time will probably be 1-3hrs. if no problems. This isn't the first system upgrade we have done, so I expect no issues.
>>> 
>>> If any of you wish to see some of the forum performance stats, I can post another message with numbers and personal observations. A quick overview of what our daily forum activity looks like:
>>> 
>>> * Serving 120-160K page views. (about 25% attributed to crawlers - a different discussion)
>>> * 6-8K mail messages generated
>>> * ~1100 User visits (signed-in)
>>> * ~200 engaged users (liked or posted)
>>> * ~500 new posts
>>> 
>>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate Droplet use)
>>> Proposed Droplet upgrade will add 3TB to our total.
>>> I don't have per-Droplet stats, but last month we transmitted ~29TB of our 25TB allowance. A $40 overage cost.
>>> 
>>> 
>>> _______________________________________________
>>> openwrt-adm mailing list
>>> openwrt-adm at lists.openwrt.org
>>> https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>> 
>> 
>> 
>> _______________________________________________
>> openwrt-adm mailing list
>> openwrt-adm at lists.openwrt.org
>> https://lists.openwrt.org/mailman/listinfo/openwrt-adm
> 
> 
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm



From champetier.etienne at gmail.com  Sat Aug 10 20:17:14 2024
From: champetier.etienne at gmail.com (Etienne Champetier)
Date: Sat, 10 Aug 2024 23:17:14 -0400
Subject: Forum host upgrade - proposal
In-Reply-To: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
Message-ID: <CAOdf3grN1hU1JRXknvjs-ayHhoG127G9yVW6x3Sgs_i__AKk6Q@mail.gmail.com>

Hi Tess,

First thanks for all the work !

Le sam. 10 ao?t 2024 ? 11:54, Ted Hess <thess at kitschensync.net> a ?crit :
>
> Hi all -
>
> Those of you who visit the forum have probably noticed its, sometimes
> extreme, sluggishness over the last several months. Since April, I have
> worked on and applied a number of attempts at tuning, cleanup and
> storage re-arrangement to try and alleviate these issues. As best as I
> can determine, we have have just simply exceeded the capacity of the
> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> The difference in cost is a "Regular Intel" vs "Premium Intel"
> processors. The Premium (latest Xeon) option will provide enhanced NVMe
> SSDs which would improve DB performance (a bit). I think it would be
> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> vs Intel - cost is the same. AMD may be slightly higher GHz.

How up to date is the wiki infrastructure page
(https://openwrt.org/infrastructure#servers) ?
Is Digital Ocean still offering some credits ?

At 48$ on DO website I see we are talking about 'basic droplets', i.e.
shared CPUs,
so the first question is how much cpu steals time do you see ?
If we see high cpu steal time, it might be better to go dedicated
(general purpose) with the same size rather than bigger.

For the SSD performance (high IO wait ?)
I have no idea if you are using the local storage for the DB and if
that would help on DO, but sharing just in case.
If TRIM is not used/can't be used (RAID card/how DO does the
virtualisation/...),
I've seen SSD crawl to 1/10 of their performance after some time.
A trick is to fill your partition free space with 0s multiple times,
if the SSD firmware is smart enough it'll do a trim for the blocks
where you put 0s.
(dd if=/dev/zero of=/bigfile status=progress; rm -f /bigfile)

To throw more power at the problem, another solution is renting bare
metal servers.
For example at OVH, If you look at a SYS-6-SSD-128, for $64.99/month you have
- AMD Epyc 7351p - 16c/32t - 2.4 GHz/2.9 GHz (same generation as DO
non premium / x8 vCPUs)
- 128GB DDR4 ECC 2400MHz (x16 mem)
- 2? 500GB SSD NVMe Soft RAID (??)
- 500Mbps unmetered internet connection

The big catches I can think of:
- you pay 1 month installation fee so you can't easily change sizes
- storage is not flexible at all
- no easy full system backup / snapshots
(I'm a happy OVH customer for years / not affiliated)

Best
Etienne

> Are there any objections / reasons why we shouldn't go ahead and do the
> upgrade. There are a couple of methods available however, the down-time
> will probably be 1-3hrs. if no problems. This isn't the first system
> upgrade we have done, so I expect no issues.
>
> If any of you wish to see some of the forum performance stats, I can
> post another message with numbers and personal observations. A quick
> overview of what our daily forum activity looks like:
>
> * Serving 120-160K page views. (about 25% attributed to crawlers - a
> different discussion)
> * 6-8K mail messages generated
> * ~1100 User visits (signed-in)
> * ~200 engaged users (liked or posted)
> * ~500 new posts
>
> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> Droplet use)
> Proposed Droplet upgrade will add 3TB to our total.
> I don't have per-Droplet stats, but last month we transmitted ~29TB of
> our 25TB allowance. A $40 overage cost.


From ardeleanalex at gmail.com  Sat Aug 10 23:00:33 2024
From: ardeleanalex at gmail.com (Alexandru Ardelean)
Date: Sun, 11 Aug 2024 09:00:33 +0300
Subject: Forum host upgrade - proposal
In-Reply-To: <CAOdf3grN1hU1JRXknvjs-ayHhoG127G9yVW6x3Sgs_i__AKk6Q@mail.gmail.com>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7> <CAOdf3grN1hU1JRXknvjs-ayHhoG127G9yVW6x3Sgs_i__AKk6Q@mail.gmail.com>
Message-ID: <CA+U=Dsq+Lc5zxwkKTyX9UGsFZ4d-qvK_x8y2HA6ihunJNbQMqQ@mail.gmail.com>

On Sun, Aug 11, 2024 at 6:18?AM Etienne Champetier
<champetier.etienne at gmail.com> wrote:
>
> Hi Tess,
>
> First thanks for all the work !
>
> Le sam. 10 ao?t 2024 ? 11:54, Ted Hess <thess at kitschensync.net> a ?crit :
> >
> > Hi all -
> >
> > Those of you who visit the forum have probably noticed its, sometimes
> > extreme, sluggishness over the last several months. Since April, I have
> > worked on and applied a number of attempts at tuning, cleanup and
> > storage re-arrangement to try and alleviate these issues. As best as I
> > can determine, we have have just simply exceeded the capacity of the
> > current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> > (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> > The difference in cost is a "Regular Intel" vs "Premium Intel"
> > processors. The Premium (latest Xeon) option will provide enhanced NVMe
> > SSDs which would improve DB performance (a bit). I think it would be
> > worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> > vs Intel - cost is the same. AMD may be slightly higher GHz.
>
> How up to date is the wiki infrastructure page
> (https://openwrt.org/infrastructure#servers) ?
> Is Digital Ocean still offering some credits ?
>
> At 48$ on DO website I see we are talking about 'basic droplets', i.e.
> shared CPUs,
> so the first question is how much cpu steals time do you see ?
> If we see high cpu steal time, it might be better to go dedicated
> (general purpose) with the same size rather than bigger.
>
> For the SSD performance (high IO wait ?)
> I have no idea if you are using the local storage for the DB and if
> that would help on DO, but sharing just in case.
> If TRIM is not used/can't be used (RAID card/how DO does the
> virtualisation/...),
> I've seen SSD crawl to 1/10 of their performance after some time.
> A trick is to fill your partition free space with 0s multiple times,
> if the SSD firmware is smart enough it'll do a trim for the blocks
> where you put 0s.
> (dd if=/dev/zero of=/bigfile status=progress; rm -f /bigfile)
>
> To throw more power at the problem, another solution is renting bare
> metal servers.
> For example at OVH, If you look at a SYS-6-SSD-128, for $64.99/month you have
> - AMD Epyc 7351p - 16c/32t - 2.4 GHz/2.9 GHz (same generation as DO
> non premium / x8 vCPUs)
> - 128GB DDR4 ECC 2400MHz (x16 mem)
> - 2? 500GB SSD NVMe Soft RAID (??)
> - 500Mbps unmetered internet connection
>
> The big catches I can think of:
> - you pay 1 month installation fee so you can't easily change sizes
> - storage is not flexible at all
> - no easy full system backup / snapshots
> (I'm a happy OVH customer for years / not affiliated)
>

I was going to reply earlier, but I wasn't around my computer, and the
GMail client won't allow for plain-text message formatting.
But to also add to Etienne's suggestion: I (personally) started seeing
that DO isn't quite the cheapest solution (as they used to be many
years ago).

OVH seems interesting (btw).
In our region, hetzner.com is popular for dedicated hosts.
And they're also offering VMs now (apparently).

I'm not saying to use hetzner.
I'm just wondering, if switching providers works (without much extra
effort), seeing as that there seem to be cheaper alternatives (that
DO) now.
And DO doesn't seem to be too cheap anymore.

> Best
> Etienne
>
> > Are there any objections / reasons why we shouldn't go ahead and do the
> > upgrade. There are a couple of methods available however, the down-time
> > will probably be 1-3hrs. if no problems. This isn't the first system
> > upgrade we have done, so I expect no issues.
> >
> > If any of you wish to see some of the forum performance stats, I can
> > post another message with numbers and personal observations. A quick
> > overview of what our daily forum activity looks like:
> >
> > * Serving 120-160K page views. (about 25% attributed to crawlers - a
> > different discussion)
> > * 6-8K mail messages generated
> > * ~1100 User visits (signed-in)
> > * ~200 engaged users (liked or posted)
> > * ~500 new posts
> >
> > Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> > Droplet use)
> > Proposed Droplet upgrade will add 3TB to our total.
> > I don't have per-Droplet stats, but last month we transmitted ~29TB of
> > our 25TB allowance. A $40 overage cost.
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm


From robimarko at gmail.com  Mon Aug 12 05:33:28 2024
From: robimarko at gmail.com (Robert Marko)
Date: Mon, 12 Aug 2024 14:33:28 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
Message-ID: <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com>

On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> wrote:
>
> Hi all -
>
> Those of you who visit the forum have probably noticed its, sometimes
> extreme, sluggishness over the last several months. Since April, I have
> worked on and applied a number of attempts at tuning, cleanup and
> storage re-arrangement to try and alleviate these issues. As best as I
> can determine, we have have just simply exceeded the capacity of the
> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> The difference in cost is a "Regular Intel" vs "Premium Intel"
> processors. The Premium (latest Xeon) option will provide enhanced NVMe
> SSDs which would improve DB performance (a bit). I think it would be
> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> vs Intel - cost is the same. AMD may be slightly higher GHz.
>
> Are there any objections / reasons why we shouldn't go ahead and do the
> upgrade. There are a couple of methods available however, the down-time
> will probably be 1-3hrs. if no problems. This isn't the first system
> upgrade we have done, so I expect no issues.
>
> If any of you wish to see some of the forum performance stats, I can
> post another message with numbers and personal observations. A quick
> overview of what our daily forum activity looks like:
>
> * Serving 120-160K page views. (about 25% attributed to crawlers - a
> different discussion)
> * 6-8K mail messages generated
> * ~1100 User visits (signed-in)
> * ~200 engaged users (liked or posted)
> * ~500 new posts
>
> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> Droplet use)
> Proposed Droplet upgrade will add 3TB to our total.
> I don't have per-Droplet stats, but last month we transmitted ~29TB of
> our 25TB allowance. A $40 overage cost.

Hi Ted,
I am for upgrading the droplet as forum is probably the main point of
interaction
in the OpenWrt community.

But, like others pointed out, we maybe should look for a more affordable host
then DO as they are quite expensive.

Regards,
Robert
>
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm


From stijn at linux-ipv6.be  Mon Aug 12 13:11:04 2024
From: stijn at linux-ipv6.be (Stijn Tintel)
Date: Mon, 12 Aug 2024 23:11:04 +0300
Subject: Forum host upgrade - proposal
In-Reply-To: <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com>
Message-ID: <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>


On 8/12/24 15:33, Robert Marko wrote:
> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> wrote:
>> Hi all -
>>
>> Those of you who visit the forum have probably noticed its, sometimes
>> extreme, sluggishness over the last several months. Since April, I have
>> worked on and applied a number of attempts at tuning, cleanup and
>> storage re-arrangement to try and alleviate these issues. As best as I
>> can determine, we have have just simply exceeded the capacity of the
>> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>> processors. The Premium (latest Xeon) option will provide enhanced NVMe
>> SSDs which would improve DB performance (a bit). I think it would be
>> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>>
>> Are there any objections / reasons why we shouldn't go ahead and do the
>> upgrade. There are a couple of methods available however, the down-time
>> will probably be 1-3hrs. if no problems. This isn't the first system
>> upgrade we have done, so I expect no issues.
>>
>> If any of you wish to see some of the forum performance stats, I can
>> post another message with numbers and personal observations. A quick
>> overview of what our daily forum activity looks like:
>>
>> * Serving 120-160K page views. (about 25% attributed to crawlers - a
>> different discussion)
>> * 6-8K mail messages generated
>> * ~1100 User visits (signed-in)
>> * ~200 engaged users (liked or posted)
>> * ~500 new posts
>>
>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>> Droplet use)
>> Proposed Droplet upgrade will add 3TB to our total.
>> I don't have per-Droplet stats, but last month we transmitted ~29TB of
>> our 25TB allowance. A $40 overage cost.
> Hi Ted,
> I am for upgrading the droplet as forum is probably the main point of
> interaction
> in the OpenWrt community.
>
> But, like others pointed out, we maybe should look for a more affordable host
> then DO as they are quite expensive.

We use DO because they sponsor us USD 5000 to use their services. If you 
look at the current costs, upgrading another droplet will cause us to 
use up the USD 5000 before they expire. Please keep that in mind before 
doing anything that will increase cost.

Stijn



From koen.vandeputte at citymesh.com  Tue Aug 13 00:22:46 2024
From: koen.vandeputte at citymesh.com (Koen Vandeputte)
Date: Tue, 13 Aug 2024 09:22:46 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com> <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
Message-ID: <CAPh3n80PcpD23jpqb7aoq5BFyuo-2ZGGBYko2VDCnJYvwavtVg@mail.gmail.com>

Hi all,

I work at Citymesh which is an ISP in Belgium.
I noticed this topic and had a chat with our datacenter lead if we can
do something to help out here.

As we are currently renewing the full server setup in Datacenter, they
are really open to see if we could provide proper (redundant)
resources for this as soon as I make an official request internally.
(30TB of traffic per month is totally no issue at all, as a single
offshore client easily consumes this in a single day.)

So if you all agree, I can file that request asap.
If it gets approved, I would bring some of you (Ted ? anyone else?) in
contact with our Datacenter team for the practical stuff.

Let me know :-)

Regards,

Koen


On Mon, Aug 12, 2024 at 10:11?PM Stijn Tintel <stijn at linux-ipv6.be> wrote:
>
>
> On 8/12/24 15:33, Robert Marko wrote:
> > On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> wrote:
> >> Hi all -
> >>
> >> Those of you who visit the forum have probably noticed its, sometimes
> >> extreme, sluggishness over the last several months. Since April, I have
> >> worked on and applied a number of attempts at tuning, cleanup and
> >> storage re-arrangement to try and alleviate these issues. As best as I
> >> can determine, we have have just simply exceeded the capacity of the
> >> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> >> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> >> The difference in cost is a "Regular Intel" vs "Premium Intel"
> >> processors. The Premium (latest Xeon) option will provide enhanced NVMe
> >> SSDs which would improve DB performance (a bit). I think it would be
> >> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> >> vs Intel - cost is the same. AMD may be slightly higher GHz.
> >>
> >> Are there any objections / reasons why we shouldn't go ahead and do the
> >> upgrade. There are a couple of methods available however, the down-time
> >> will probably be 1-3hrs. if no problems. This isn't the first system
> >> upgrade we have done, so I expect no issues.
> >>
> >> If any of you wish to see some of the forum performance stats, I can
> >> post another message with numbers and personal observations. A quick
> >> overview of what our daily forum activity looks like:
> >>
> >> * Serving 120-160K page views. (about 25% attributed to crawlers - a
> >> different discussion)
> >> * 6-8K mail messages generated
> >> * ~1100 User visits (signed-in)
> >> * ~200 engaged users (liked or posted)
> >> * ~500 new posts
> >>
> >> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> >> Droplet use)
> >> Proposed Droplet upgrade will add 3TB to our total.
> >> I don't have per-Droplet stats, but last month we transmitted ~29TB of
> >> our 25TB allowance. A $40 overage cost.
> > Hi Ted,
> > I am for upgrading the droplet as forum is probably the main point of
> > interaction
> > in the OpenWrt community.
> >
> > But, like others pointed out, we maybe should look for a more affordable host
> > then DO as they are quite expensive.
>
> We use DO because they sponsor us USD 5000 to use their services. If you
> look at the current costs, upgrading another droplet will cause us to
> use up the USD 5000 before they expire. Please keep that in mind before
> doing anything that will increase cost.
>
> Stijn
>
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm


From ardeleanalex at gmail.com  Tue Aug 13 00:37:03 2024
From: ardeleanalex at gmail.com (Alexandru Ardelean)
Date: Tue, 13 Aug 2024 10:37:03 +0300
Subject: Forum host upgrade - proposal
In-Reply-To: <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com> <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
Message-ID: <CA+U=DspgxFeE_A_VdvOgf-uxciToUXw7JmcC9HfQoEcP7cJ0KQ@mail.gmail.com>

On Mon, Aug 12, 2024 at 11:12?PM Stijn Tintel <stijn at linux-ipv6.be> wrote:
>
>
> On 8/12/24 15:33, Robert Marko wrote:
> > On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> wrote:
> >> Hi all -
> >>
> >> Those of you who visit the forum have probably noticed its, sometimes
> >> extreme, sluggishness over the last several months. Since April, I have
> >> worked on and applied a number of attempts at tuning, cleanup and
> >> storage re-arrangement to try and alleviate these issues. As best as I
> >> can determine, we have have just simply exceeded the capacity of the
> >> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
> >> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
> >> The difference in cost is a "Regular Intel" vs "Premium Intel"
> >> processors. The Premium (latest Xeon) option will provide enhanced NVMe
> >> SSDs which would improve DB performance (a bit). I think it would be
> >> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
> >> vs Intel - cost is the same. AMD may be slightly higher GHz.
> >>
> >> Are there any objections / reasons why we shouldn't go ahead and do the
> >> upgrade. There are a couple of methods available however, the down-time
> >> will probably be 1-3hrs. if no problems. This isn't the first system
> >> upgrade we have done, so I expect no issues.
> >>
> >> If any of you wish to see some of the forum performance stats, I can
> >> post another message with numbers and personal observations. A quick
> >> overview of what our daily forum activity looks like:
> >>
> >> * Serving 120-160K page views. (about 25% attributed to crawlers - a
> >> different discussion)
> >> * 6-8K mail messages generated
> >> * ~1100 User visits (signed-in)
> >> * ~200 engaged users (liked or posted)
> >> * ~500 new posts
> >>
> >> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> >> Droplet use)
> >> Proposed Droplet upgrade will add 3TB to our total.
> >> I don't have per-Droplet stats, but last month we transmitted ~29TB of
> >> our 25TB allowance. A $40 overage cost.
> > Hi Ted,
> > I am for upgrading the droplet as forum is probably the main point of
> > interaction
> > in the OpenWrt community.
> >
> > But, like others pointed out, we maybe should look for a more affordable host
> > then DO as they are quite expensive.
>
> We use DO because they sponsor us USD 5000 to use their services. If you
> look at the current costs, upgrading another droplet will cause us to
> use up the USD 5000 before they expire. Please keep that in mind before
> doing anything that will increase cost.

I did not know this.
In light of this new info, I will retract my suggestion.

Thanks
Alex

>
> Stijn
>
>
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm


From hauke at hauke-m.de  Wed Aug 14 16:05:24 2024
From: hauke at hauke-m.de (Hauke Mehrtens)
Date: Thu, 15 Aug 2024 01:05:24 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com>
 <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
Message-ID: <71b61503-6c81-4ef5-89a6-c40264f9058e@hauke-m.de>

On 8/12/24 22:11, Stijn Tintel wrote:
> 
> On 8/12/24 15:33, Robert Marko wrote:
>> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> wrote:
>>> Hi all -
>>>
>>> Those of you who visit the forum have probably noticed its, sometimes
>>> extreme, sluggishness over the last several months. Since April, I have
>>> worked on and applied a number of attempts at tuning, cleanup and
>>> storage re-arrangement to try and alleviate these issues. As best as I
>>> can determine, we have have just simply exceeded the capacity of the
>>> current Droplet CPU and RAM. To upgrade to a system with 2X CPU/RAM
>>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to $96-112/mo.
>>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>>> processors. The Premium (latest Xeon) option will provide enhanced NVMe
>>> SSDs which would improve DB performance (a bit). I think it would be
>>> worth $112/mo for maximal impact on the responsiveness of the forum. AMD
>>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>>>
>>> Are there any objections / reasons why we shouldn't go ahead and do the
>>> upgrade. There are a couple of methods available however, the down-time
>>> will probably be 1-3hrs. if no problems. This isn't the first system
>>> upgrade we have done, so I expect no issues.
>>>
>>> If any of you wish to see some of the forum performance stats, I can
>>> post another message with numbers and personal observations. A quick
>>> overview of what our daily forum activity looks like:
>>>
>>> * Serving 120-160K page views. (about 25% attributed to crawlers - a
>>> different discussion)
>>> * 6-8K mail messages generated
>>> * ~1100 User visits (signed-in)
>>> * ~200 engaged users (liked or posted)
>>> * ~500 new posts
>>>
>>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>>> Droplet use)
>>> Proposed Droplet upgrade will add 3TB to our total.
>>> I don't have per-Droplet stats, but last month we transmitted ~29TB of
>>> our 25TB allowance. A $40 overage cost.
>> Hi Ted,
>> I am for upgrading the droplet as forum is probably the main point of
>> interaction
>> in the OpenWrt community.
>>
>> But, like others pointed out, we maybe should look for a more 
>> affordable host
>> then DO as they are quite expensive.
> 
> We use DO because they sponsor us USD 5000 to use their services. If you 
> look at the current costs, upgrading another droplet will cause us to 
> use up the USD 5000 before they expire. Please keep that in mind before 
> doing anything that will increase cost.
> 
> Stijn

Hi,

I think it is fine if we invest some money into the server for the 
forum. From my point of view it is fine to upgrade the server from 
4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.

We should consider to move some resources to other providers like 
Hetzner, but we can do this later.


Hauke


From thess at kitschensync.net  Fri Aug 16 12:15:52 2024
From: thess at kitschensync.net (Ted Hess)
Date: Fri, 16 Aug 2024 19:15:52 +0000
Subject: Forum host upgrade - proposal
Message-ID: <em0ee5dcb3-79a3-4204-91ea-f9156c46361e@twonky7>

On 8/16/2024 2:37:31 PM, "Koen Vandeputte" 
<koen.vandeputte at citymesh.com> wrote:

>
>
>Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
>>On 8/12/24 22:11, Stijn Tintel wrote:
>> >
>> > On 8/12/24 15:33, Robert Marko wrote:
>> >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net> 
>>wrote:
>> >>> Hi all -
>> >>>
>> >>> Those of you who visit the forum have probably noticed its, 
>>sometimes
>> >>> extreme, sluggishness over the last several months. Since April, I 
>>have
>> >>> worked on and applied a number of attempts at tuning, cleanup and
>> >>> storage re-arrangement to try and alleviate these issues. As best 
>>as I
>> >>> can determine, we have have just simply exceeded the capacity of 
>>the
>> >>> current Droplet CPU and RAM. To upgrade to a system with 2X 
>>CPU/RAM
>> >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to 
>>$96-112/mo.
>> >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>> >>> processors. The Premium (latest Xeon) option will provide enhanced 
>>NVMe
>> >>> SSDs which would improve DB performance (a bit). I think it would 
>>be
>> >>> worth $112/mo for maximal impact on the responsiveness of the 
>>forum. AMD
>> >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>> >>>
>> >>> Are there any objections / reasons why we shouldn't go ahead and 
>>do the
>> >>> upgrade. There are a couple of methods available however, the 
>>down-time
>> >>> will probably be 1-3hrs. if no problems. This isn't the first 
>>system
>> >>> upgrade we have done, so I expect no issues.
>> >>>
>> >>> If any of you wish to see some of the forum performance stats, I 
>>can
>> >>> post another message with numbers and personal observations. A 
>>quick
>> >>> overview of what our daily forum activity looks like:
>> >>>
>> >>> * Serving 120-160K page views. (about 25% attributed to crawlers - 
>>a
>> >>> different discussion)
>> >>> * 6-8K mail messages generated
>> >>> * ~1100 User visits (signed-in)
>> >>> * ~200 engaged users (liked or posted)
>> >>> * ~500 new posts
>> >>>
>> >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>> >>> Droplet use)
>> >>> Proposed Droplet upgrade will add 3TB to our total.
>> >>> I don't have per-Droplet stats, but last month we transmitted 
>>~29TB of
>> >>> our 25TB allowance. A $40 overage cost.
>> >> Hi Ted,
>> >> I am for upgrading the droplet as forum is probably the main point 
>>of
>> >> interaction
>> >> in the OpenWrt community.
>> >>
>> >> But, like others pointed out, we maybe should look for a more
>> >> affordable host
>> >> then DO as they are quite expensive.
>> >
>> > We use DO because they sponsor us USD 5000 to use their services. If 
>>you
>> > look at the current costs, upgrading another droplet will cause us 
>>to
>> > use up the USD 5000 before they expire. Please keep that in mind 
>>before
>> > doing anything that will increase cost.
>> >
>> > Stijn
>>
>>Hi,
>>
>>I think it is fine if we invest some money into the server for the
>>forum. From my point of view it is fine to upgrade the server from
>>4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
>>
>>We should consider to move some resources to other providers like
>>Hetzner, but we can do this later.
>>
>>
>>Hauke
>>
>>_______________________________________________
>>openwrt-adm mailing list
>>openwrt-adm at lists.openwrt.org
>>https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>
>Hi all,
>
>Is there any interest in my proposal for hosting the forum at Citymesh? 
>(Free of charge)
>Datacenter team has given a Green Light internally.
>
Koen -

Thanks for the offer. From what I can tell, this is not Citymesh's core 
business. And, as Stijn pointed out, we exist at DO essentially for 
free. What we have at DO for services beyond just a system + network are 
backups, monitoring, easy (re-)configuring, local private network 
access, hard console access (for when it don't startup). fixed IP 
addresses and DNS management. They also host our mailer (MTA, SMTP & 
POP3) which is integral with the forum.

I would prefer not to move out of DO unless we have to for reasons of 
cost and/or performance - not fun.

/ted



From stijn at linux-ipv6.be  Fri Aug 16 12:48:58 2024
From: stijn at linux-ipv6.be (Stijn Tintel)
Date: Fri, 16 Aug 2024 22:48:58 +0300
Subject: Forum host upgrade - proposal
In-Reply-To: <CAPh3n80SDdEBH3ruc0wSts6Y9Zmoci++gK6Ye48GK2mG2kAzQA@mail.gmail.com>
References: <emfb6fcad3-84c7-48e0-8a00-0e6ee9282827@twonky7>
 <CAOX2RU5zFj2xQo7LDL24juMq9T5nLba2nRa06c5yCgTwjfU==A@mail.gmail.com>
 <1cb1c6e9-8b40-4451-be1e-2df65dc6a451@linux-ipv6.be>
 <71b61503-6c81-4ef5-89a6-c40264f9058e@hauke-m.de>
 <CAPh3n80SDdEBH3ruc0wSts6Y9Zmoci++gK6Ye48GK2mG2kAzQA@mail.gmail.com>
Message-ID: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>

Resend because someone decided to send HTML email which tricked my email 
client to sending HTML email as well, which resulted in my response 
being rejected.

Please people, stop sending HTML email, and while at it, stop top-posting.

On 16/08/2024 21:37, Koen Vandeputte wrote:
>
>
> Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
>
>     On 8/12/24 22:11, Stijn Tintel wrote:
>     >
>     > On 8/12/24 15:33, Robert Marko wrote:
>     >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
>     wrote:
>     >>> Hi all -
>     >>>
>     >>> Those of you who visit the forum have probably noticed its,
>     sometimes
>     >>> extreme, sluggishness over the last several months. Since
>     April, I have
>     >>> worked on and applied a number of attempts at tuning, cleanup and
>     >>> storage re-arrangement to try and alleviate these issues. As
>     best as I
>     >>> can determine, we have have just simply exceeded the capacity
>     of the
>     >>> current Droplet CPU and RAM. To upgrade to a system with 2X
>     CPU/RAM
>     >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
>     $96-112/mo.
>     >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>     >>> processors. The Premium (latest Xeon) option will provide
>     enhanced NVMe
>     >>> SSDs which would improve DB performance (a bit). I think it
>     would be
>     >>> worth $112/mo for maximal impact on the responsiveness of the
>     forum. AMD
>     >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>     >>>
>     >>> Are there any objections / reasons why we shouldn't go ahead
>     and do the
>     >>> upgrade. There are a couple of methods available however, the
>     down-time
>     >>> will probably be 1-3hrs. if no problems. This isn't the first
>     system
>     >>> upgrade we have done, so I expect no issues.
>     >>>
>     >>> If any of you wish to see some of the forum performance stats,
>     I can
>     >>> post another message with numbers and personal observations. A
>     quick
>     >>> overview of what our daily forum activity looks like:
>     >>>
>     >>> * Serving 120-160K page views. (about 25% attributed to
>     crawlers - a
>     >>> different discussion)
>     >>> * 6-8K mail messages generated
>     >>> * ~1100 User visits (signed-in)
>     >>> * ~200 engaged users (liked or posted)
>     >>> * ~500 new posts
>     >>>
>     >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>     >>> Droplet use)
>     >>> Proposed Droplet upgrade will add 3TB to our total.
>     >>> I don't have per-Droplet stats, but last month we transmitted
>     ~29TB of
>     >>> our 25TB allowance. A $40 overage cost.
>     >> Hi Ted,
>     >> I am for upgrading the droplet as forum is probably the main
>     point of
>     >> interaction
>     >> in the OpenWrt community.
>     >>
>     >> But, like others pointed out, we maybe should look for a more
>     >> affordable host
>     >> then DO as they are quite expensive.
>     >
>     > We use DO because they sponsor us USD 5000 to use their
>     services. If you
>     > look at the current costs, upgrading another droplet will cause
>     us to
>     > use up the USD 5000 before they expire. Please keep that in mind
>     before
>     > doing anything that will increase cost.
>     >
>     > Stijn
>
>     Hi,
>
>     I think it is fine if we invest some money into the server for the
>     forum. From my point of view it is fine to upgrade the server from
>     4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
>
>     We should consider to move some resources to other providers like
>     Hetzner, but we can do this later.
>
>
>     Hauke
>
>     _______________________________________________
>     openwrt-adm mailing list
>     openwrt-adm at lists.openwrt.org
>     https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>
>
> Hi all,
>
> Is there any interest in my proposal for hosting the forum at 
> Citymesh? (Free of charge)
> Datacenter team has given a Green Light internally.
>
> Thanks,
>
> Koen (Mobile)
>
>
Hi Koen,

While I appreciate the offer, I am hesitant to accept it. I remember one 
of the main driving factors of the LEDE fork all too well. One of them 
was the downtime of key OpenWrt infrastructure, with the inability of 
all but one OpenWrt team members to solve said downtime, due to the rest 
not having access. This is something we absolutely have to avoid from 
happening again in the future. That said, I would be open to accepting 
the offer, if the following conditions are met:

At least 3 (preferably 5) OpenWrt team members who are currently 
involved in maintaining the OpenWrt infrastructure should have full 
access. As to full access, this includes the ability to call someone at 
any given time of day to recover something that is broken.

Keeping in mind that cyber attacks are currently happening everywhere, 
phishing-proof 2FA must be mandatory. In my opinion, the only standard 
that provides this, is FIDO2 hardware keys.

A signed statement from a major stakeholder at Citymesh, that if a 
decision is made to retract the offer, we are guaranteed to have at 
least six months to migrate everything elsewhere.

I know I am harsh on this, but we should learn from the past.

Best,
Stijn


From mail at aparcar.org  Sat Aug 17 01:13:18 2024
From: mail at aparcar.org (Paul Spooren)
Date: Sat, 17 Aug 2024 10:13:18 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>
References: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>
Message-ID: <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>

Hi,

> On 16. Aug 2024, at 21:49, Stijn Tintel <stijn at linux-ipv6.be> wrote:
> 
> ?Resend because someone decided to send HTML email which tricked my email client to sending HTML email as well, which resulted in my response being rejected.
> 
> Please people, stop sending HTML email, and while at it, stop top-posting.
> 
>> On 16/08/2024 21:37, Koen Vandeputte wrote:
>> 
>> 
>> Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
>> 
>>    On 8/12/24 22:11, Stijn Tintel wrote:
>>    >
>>    > On 8/12/24 15:33, Robert Marko wrote:
>>    >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
>>    wrote:
>>    >>> Hi all -
>>    >>>
>>    >>> Those of you who visit the forum have probably noticed its,
>>    sometimes
>>    >>> extreme, sluggishness over the last several months. Since
>>    April, I have
>>    >>> worked on and applied a number of attempts at tuning, cleanup and
>>    >>> storage re-arrangement to try and alleviate these issues. As
>>    best as I
>>    >>> can determine, we have have just simply exceeded the capacity
>>    of the
>>    >>> current Droplet CPU and RAM. To upgrade to a system with 2X
>>    CPU/RAM
>>    >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
>>    $96-112/mo.
>>    >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>>    >>> processors. The Premium (latest Xeon) option will provide
>>    enhanced NVMe
>>    >>> SSDs which would improve DB performance (a bit). I think it
>>    would be
>>    >>> worth $112/mo for maximal impact on the responsiveness of the
>>    forum. AMD
>>    >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>>    >>>
>>    >>> Are there any objections / reasons why we shouldn't go ahead
>>    and do the
>>    >>> upgrade. There are a couple of methods available however, the
>>    down-time
>>    >>> will probably be 1-3hrs. if no problems. This isn't the first
>>    system
>>    >>> upgrade we have done, so I expect no issues.
>>    >>>
>>    >>> If any of you wish to see some of the forum performance stats,
>>    I can
>>    >>> post another message with numbers and personal observations. A
>>    quick
>>    >>> overview of what our daily forum activity looks like:
>>    >>>
>>    >>> * Serving 120-160K page views. (about 25% attributed to
>>    crawlers - a
>>    >>> different discussion)
>>    >>> * 6-8K mail messages generated
>>    >>> * ~1100 User visits (signed-in)
>>    >>> * ~200 engaged users (liked or posted)
>>    >>> * ~500 new posts
>>    >>>
>>    >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>>    >>> Droplet use)
>>    >>> Proposed Droplet upgrade will add 3TB to our total.
>>    >>> I don't have per-Droplet stats, but last month we transmitted
>>    ~29TB of
>>    >>> our 25TB allowance. A $40 overage cost.
>>    >> Hi Ted,
>>    >> I am for upgrading the droplet as forum is probably the main
>>    point of
>>    >> interaction
>>    >> in the OpenWrt community.
>>    >>
>>    >> But, like others pointed out, we maybe should look for a more
>>    >> affordable host
>>    >> then DO as they are quite expensive.
>>    >
>>    > We use DO because they sponsor us USD 5000 to use their
>>    services. If you
>>    > look at the current costs, upgrading another droplet will cause
>>    us to
>>    > use up the USD 5000 before they expire. Please keep that in mind
>>    before
>>    > doing anything that will increase cost.
>>    >
>>    > Stijn
>> 
>>    Hi,
>> 
>>    I think it is fine if we invest some money into the server for the
>>    forum. From my point of view it is fine to upgrade the server from
>>    4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
>> 
>>    We should consider to move some resources to other providers like
>>    Hetzner, but we can do this later.
>> 
>> 
>>    Hauke
>> 
>>    _______________________________________________
>>    openwrt-adm mailing list
>>    openwrt-adm at lists.openwrt.org
>>    https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>> 
>> 
>> Hi all,
>> 
>> Is there any interest in my proposal for hosting the forum at Citymesh? (Free of charge)
>> Datacenter team has given a Green Light internally.
>> 
>> Thanks,
>> 
>> Koen (Mobile)
>> 
>> 
> Hi Koen,
> 
> While I appreciate the offer, I am hesitant to accept it. I remember one of the main driving factors of the LEDE fork all too well. One of them was the downtime of key OpenWrt infrastructure, with the inability of all but one OpenWrt team members to solve said downtime, due to the rest not having access. This is something we absolutely have to avoid from happening again in the future. That said, I would be open to accepting the offer, if the following conditions are met:
> 
> At least 3 (preferably 5) OpenWrt team members who are currently involved in maintaining the OpenWrt infrastructure should have full access. As to full access, this includes the ability to call someone at any given time of day to recover something that is broken.
> 
> Keeping in mind that cyber attacks are currently happening everywhere, phishing-proof 2FA must be mandatory. In my opinion, the only standard that provides this, is FIDO2 hardware keys.
> 
> A signed statement from a major stakeholder at Citymesh, that if a decision is made to retract the offer, we are guaranteed to have at least six months to migrate everything elsewhere.
> 
> I know I am harsh on this, but we should learn from the past.

I did not follow the whole thread but when it comes to having a more powerful machine at DO, I could migrate away the ASU server to another provider. It?s less critical infrastructure and rather costly due to CPU and storage requirements.

It?s rather trivial to setup so I could move it within days. Please let me know if that would be of help.

Best,
Paul

From koen.vandeputte at citymesh.com  Tue Aug 20 08:48:37 2024
From: koen.vandeputte at citymesh.com (Koen Vandeputte)
Date: Tue, 20 Aug 2024 17:48:37 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>
References: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be> <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>
Message-ID: <CAPh3n82RJNWAX1y0m3Sd3uHRhhp62RNZFYQXkhEZLffRwXbB0A@mail.gmail.com>

On Sat, Aug 17, 2024 at 10:13?AM Paul Spooren <mail at aparcar.org> wrote:
>
> Hi,
>
> > On 16. Aug 2024, at 21:49, Stijn Tintel <stijn at linux-ipv6.be> wrote:
> >
> > ?Resend because someone decided to send HTML email which tricked my email client to sending HTML email as well, which resulted in my response being rejected.
> >
> > Please people, stop sending HTML email, and while at it, stop top-posting.
> >
> >> On 16/08/2024 21:37, Koen Vandeputte wrote:
> >>
> >>
> >> Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
> >>
> >>    On 8/12/24 22:11, Stijn Tintel wrote:
> >>    >
> >>    > On 8/12/24 15:33, Robert Marko wrote:
> >>    >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
> >>    wrote:
> >>    >>> Hi all -
> >>    >>>
> >>    >>> Those of you who visit the forum have probably noticed its,
> >>    sometimes
> >>    >>> extreme, sluggishness over the last several months. Since
> >>    April, I have
> >>    >>> worked on and applied a number of attempts at tuning, cleanup and
> >>    >>> storage re-arrangement to try and alleviate these issues. As
> >>    best as I
> >>    >>> can determine, we have have just simply exceeded the capacity
> >>    of the
> >>    >>> current Droplet CPU and RAM. To upgrade to a system with 2X
> >>    CPU/RAM
> >>    >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
> >>    $96-112/mo.
> >>    >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
> >>    >>> processors. The Premium (latest Xeon) option will provide
> >>    enhanced NVMe
> >>    >>> SSDs which would improve DB performance (a bit). I think it
> >>    would be
> >>    >>> worth $112/mo for maximal impact on the responsiveness of the
> >>    forum. AMD
> >>    >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
> >>    >>>
> >>    >>> Are there any objections / reasons why we shouldn't go ahead
> >>    and do the
> >>    >>> upgrade. There are a couple of methods available however, the
> >>    down-time
> >>    >>> will probably be 1-3hrs. if no problems. This isn't the first
> >>    system
> >>    >>> upgrade we have done, so I expect no issues.
> >>    >>>
> >>    >>> If any of you wish to see some of the forum performance stats,
> >>    I can
> >>    >>> post another message with numbers and personal observations. A
> >>    quick
> >>    >>> overview of what our daily forum activity looks like:
> >>    >>>
> >>    >>> * Serving 120-160K page views. (about 25% attributed to
> >>    crawlers - a
> >>    >>> different discussion)
> >>    >>> * 6-8K mail messages generated
> >>    >>> * ~1100 User visits (signed-in)
> >>    >>> * ~200 engaged users (liked or posted)
> >>    >>> * ~500 new posts
> >>    >>>
> >>    >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> >>    >>> Droplet use)
> >>    >>> Proposed Droplet upgrade will add 3TB to our total.
> >>    >>> I don't have per-Droplet stats, but last month we transmitted
> >>    ~29TB of
> >>    >>> our 25TB allowance. A $40 overage cost.
> >>    >> Hi Ted,
> >>    >> I am for upgrading the droplet as forum is probably the main
> >>    point of
> >>    >> interaction
> >>    >> in the OpenWrt community.
> >>    >>
> >>    >> But, like others pointed out, we maybe should look for a more
> >>    >> affordable host
> >>    >> then DO as they are quite expensive.
> >>    >
> >>    > We use DO because they sponsor us USD 5000 to use their
> >>    services. If you
> >>    > look at the current costs, upgrading another droplet will cause
> >>    us to
> >>    > use up the USD 5000 before they expire. Please keep that in mind
> >>    before
> >>    > doing anything that will increase cost.
> >>    >
> >>    > Stijn
> >>
> >>    Hi,
> >>
> >>    I think it is fine if we invest some money into the server for the
> >>    forum. From my point of view it is fine to upgrade the server from
> >>    4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
> >>
> >>    We should consider to move some resources to other providers like
> >>    Hetzner, but we can do this later.
> >>
> >>
> >>    Hauke
> >>
> >>    _______________________________________________
> >>    openwrt-adm mailing list
> >>    openwrt-adm at lists.openwrt.org
> >>    https://lists.openwrt.org/mailman/listinfo/openwrt-adm
> >>
> >>
> >> Hi all,
> >>
> >> Is there any interest in my proposal for hosting the forum at Citymesh? (Free of charge)
> >> Datacenter team has given a Green Light internally.
> >>
> >> Thanks,
> >>
> >> Koen (Mobile)
> >>
> >>
> > Hi Koen,
> >

> > While I appreciate the offer, I am hesitant to accept it. I remember one of the main driving factors of the LEDE fork all too well. One of them was the downtime of key OpenWrt infrastructure, with the inability of all but one OpenWrt team members to solve said downtime, due to the rest not having access. This is something we absolutely have to avoid from happening again in the future. That said, I would be open to accepting the offer, if the following conditions are met:
> >
> > At least 3 (preferably 5) OpenWrt team members who are currently involved in maintaining the OpenWrt infrastructure should have full access. As to full access, this includes the ability to call someone at any given time of day to recover something that is broken.
> >
> > Keeping in mind that cyber attacks are currently happening everywhere, phishing-proof 2FA must be mandatory. In my opinion, the only standard that provides this, is FIDO2 hardware keys.
> >
> > A signed statement from a major stakeholder at Citymesh, that if a decision is made to retract the offer, we are guaranteed to have at least six months to migrate everything elsewhere.
> >
> > I know I am harsh on this, but we should learn from the past.
>
> I did not follow the whole thread but when it comes to having a more powerful machine at DO, I could migrate away the ASU server to another provider. It?s less critical infrastructure and rather costly due to CPU and storage requirements.
>
> It?s rather trivial to setup so I could move it within days. Please let me know if that would be of help.
>
> Best,
> Paul

Hi Stijn & all,

I totally understand your arguments and fully respect the requirements.

I've forwarded this internally to the right people to check your requirements
and will come back to you on that.

If this would be rejected for some reason, maybe we can aid in
different needs which are less mission critical in the future. :-)

Thanks,

Koen


From koen.vandeputte at citymesh.com  Tue Aug 20 09:03:20 2024
From: koen.vandeputte at citymesh.com (Koen Vandeputte)
Date: Tue, 20 Aug 2024 18:03:20 +0200
Subject: Forum host upgrade - proposal
In-Reply-To: <CAPh3n82RJNWAX1y0m3Sd3uHRhhp62RNZFYQXkhEZLffRwXbB0A@mail.gmail.com>
References: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>
 <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org> <CAPh3n82RJNWAX1y0m3Sd3uHRhhp62RNZFYQXkhEZLffRwXbB0A@mail.gmail.com>
Message-ID: <CAPh3n81VHCKbuHzviJANUdaTdBfSnErqT=ijGZhdOPq_5OGkCg@mail.gmail.com>

On Tue, Aug 20, 2024 at 5:48?PM Koen Vandeputte
<koen.vandeputte at citymesh.com> wrote:
>
> On Sat, Aug 17, 2024 at 10:13?AM Paul Spooren <mail at aparcar.org> wrote:
> >
> > Hi,
> >
> > > On 16. Aug 2024, at 21:49, Stijn Tintel <stijn at linux-ipv6.be> wrote:
> > >
> > > ?Resend because someone decided to send HTML email which tricked my email client to sending HTML email as well, which resulted in my response being rejected.
> > >
> > > Please people, stop sending HTML email, and while at it, stop top-posting.
> > >
> > >> On 16/08/2024 21:37, Koen Vandeputte wrote:
> > >>
> > >>
> > >> Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
> > >>
> > >>    On 8/12/24 22:11, Stijn Tintel wrote:
> > >>    >
> > >>    > On 8/12/24 15:33, Robert Marko wrote:
> > >>    >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
> > >>    wrote:
> > >>    >>> Hi all -
> > >>    >>>
> > >>    >>> Those of you who visit the forum have probably noticed its,
> > >>    sometimes
> > >>    >>> extreme, sluggishness over the last several months. Since
> > >>    April, I have
> > >>    >>> worked on and applied a number of attempts at tuning, cleanup and
> > >>    >>> storage re-arrangement to try and alleviate these issues. As
> > >>    best as I
> > >>    >>> can determine, we have have just simply exceeded the capacity
> > >>    of the
> > >>    >>> current Droplet CPU and RAM. To upgrade to a system with 2X
> > >>    CPU/RAM
> > >>    >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
> > >>    $96-112/mo.
> > >>    >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
> > >>    >>> processors. The Premium (latest Xeon) option will provide
> > >>    enhanced NVMe
> > >>    >>> SSDs which would improve DB performance (a bit). I think it
> > >>    would be
> > >>    >>> worth $112/mo for maximal impact on the responsiveness of the
> > >>    forum. AMD
> > >>    >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
> > >>    >>>
> > >>    >>> Are there any objections / reasons why we shouldn't go ahead
> > >>    and do the
> > >>    >>> upgrade. There are a couple of methods available however, the
> > >>    down-time
> > >>    >>> will probably be 1-3hrs. if no problems. This isn't the first
> > >>    system
> > >>    >>> upgrade we have done, so I expect no issues.
> > >>    >>>
> > >>    >>> If any of you wish to see some of the forum performance stats,
> > >>    I can
> > >>    >>> post another message with numbers and personal observations. A
> > >>    quick
> > >>    >>> overview of what our daily forum activity looks like:
> > >>    >>>
> > >>    >>> * Serving 120-160K page views. (about 25% attributed to
> > >>    crawlers - a
> > >>    >>> different discussion)
> > >>    >>> * 6-8K mail messages generated
> > >>    >>> * ~1100 User visits (signed-in)
> > >>    >>> * ~200 engaged users (liked or posted)
> > >>    >>> * ~500 new posts
> > >>    >>>
> > >>    >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
> > >>    >>> Droplet use)
> > >>    >>> Proposed Droplet upgrade will add 3TB to our total.
> > >>    >>> I don't have per-Droplet stats, but last month we transmitted
> > >>    ~29TB of
> > >>    >>> our 25TB allowance. A $40 overage cost.
> > >>    >> Hi Ted,
> > >>    >> I am for upgrading the droplet as forum is probably the main
> > >>    point of
> > >>    >> interaction
> > >>    >> in the OpenWrt community.
> > >>    >>
> > >>    >> But, like others pointed out, we maybe should look for a more
> > >>    >> affordable host
> > >>    >> then DO as they are quite expensive.
> > >>    >
> > >>    > We use DO because they sponsor us USD 5000 to use their
> > >>    services. If you
> > >>    > look at the current costs, upgrading another droplet will cause
> > >>    us to
> > >>    > use up the USD 5000 before they expire. Please keep that in mind
> > >>    before
> > >>    > doing anything that will increase cost.
> > >>    >
> > >>    > Stijn
> > >>
> > >>    Hi,
> > >>
> > >>    I think it is fine if we invest some money into the server for the
> > >>    forum. From my point of view it is fine to upgrade the server from
> > >>    4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
> > >>
> > >>    We should consider to move some resources to other providers like
> > >>    Hetzner, but we can do this later.
> > >>
> > >>
> > >>    Hauke
> > >>
> > >>    _______________________________________________
> > >>    openwrt-adm mailing list
> > >>    openwrt-adm at lists.openwrt.org
> > >>    https://lists.openwrt.org/mailman/listinfo/openwrt-adm
> > >>
> > >>
> > >> Hi all,
> > >>
> > >> Is there any interest in my proposal for hosting the forum at Citymesh? (Free of charge)
> > >> Datacenter team has given a Green Light internally.
> > >>
> > >> Thanks,
> > >>
> > >> Koen (Mobile)
> > >>
> > >>
> > > Hi Koen,
> > >
>
> > > While I appreciate the offer, I am hesitant to accept it. I remember one of the main driving factors of the LEDE fork all too well. One of them was the downtime of key OpenWrt infrastructure, with the inability of all but one OpenWrt team members to solve said downtime, due to the rest not having access. This is something we absolutely have to avoid from happening again in the future. That said, I would be open to accepting the offer, if the following conditions are met:
> > >
> > > At least 3 (preferably 5) OpenWrt team members who are currently involved in maintaining the OpenWrt infrastructure should have full access. As to full access, this includes the ability to call someone at any given time of day to recover something that is broken.
> > >
> > > Keeping in mind that cyber attacks are currently happening everywhere, phishing-proof 2FA must be mandatory. In my opinion, the only standard that provides this, is FIDO2 hardware keys.
> > >
> > > A signed statement from a major stakeholder at Citymesh, that if a decision is made to retract the offer, we are guaranteed to have at least six months to migrate everything elsewhere.
> > >
> > > I know I am harsh on this, but we should learn from the past.
> >
> > I did not follow the whole thread but when it comes to having a more powerful machine at DO, I could migrate away the ASU server to another provider. It?s less critical infrastructure and rather costly due to CPU and storage requirements.
> >
> > It?s rather trivial to setup so I could move it within days. Please let me know if that would be of help.
> >
> > Best,
> > Paul
>
> Hi Stijn & all,
>
> I totally understand your arguments and fully respect the requirements.
>
> I've forwarded this internally to the right people to check your requirements
> and will come back to you on that.
>
> If this would be rejected for some reason, maybe we can aid in
> different needs which are less mission critical in the future. :-)
>
> Thanks,
>
> Koen


Hi all,

And I already got a reply back internally ..

Please clearly state following requirements:

- Network: estimates of current throughput + monthly volume
- Disk space currently required
- CPU (speed + cores)
- RAM (amount)

Stijn,

Please provide more details about this part:
- Keeping in mind that cyber attacks are currently happening
everywhere, phishing-proof 2FA must be mandatory. In my opinion, the
only standard that provides this, is FIDO2 hardware keys.

Your other requirements are fine and reasonable.
Communication would also happen with the DC team and/or support
directly, not through me ofc to avoid single point of failure.


If the resource specifications are accepted, I would propose to loop
some of you in with the DC team for further discussion if that's ok.


Regards,

Koen


From thess at kitschensync.net  Tue Aug 20 11:52:58 2024
From: thess at kitschensync.net (Ted Hess)
Date: Tue, 20 Aug 2024 18:52:58 +0000
Subject: Forum host upgrade - proposal
In-Reply-To: <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>
References: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>
 <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>
Message-ID: <em6066cb92-71a3-4793-9371-d4dbe7b7b6ea@twonky7>



On 8/17/2024 4:13:18 AM, "Paul Spooren" <mail at aparcar.org> wrote:

>Hi,
>
>>  On 16. Aug 2024, at 21:49, Stijn Tintel <stijn at linux-ipv6.be> wrote:
>>
>>  ?Resend because someone decided to send HTML email which tricked my email client to sending HTML email as well, which resulted in my response being rejected.
>>
>>  Please people, stop sending HTML email, and while at it, stop top-posting.
>>
>>>  On 16/08/2024 21:37, Koen Vandeputte wrote:
>>>
>>>
>>>  Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
>>>
>>>     On 8/12/24 22:11, Stijn Tintel wrote:
>>>     >
>>>     > On 8/12/24 15:33, Robert Marko wrote:
>>>     >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
>>>     wrote:
>>>     >>> Hi all -
>>>     >>>
>>>     >>> Those of you who visit the forum have probably noticed its,
>>>     sometimes
>>>     >>> extreme, sluggishness over the last several months. Since
>>>     April, I have
>>>     >>> worked on and applied a number of attempts at tuning, cleanup and
>>>     >>> storage re-arrangement to try and alleviate these issues. As
>>>     best as I
>>>     >>> can determine, we have have just simply exceeded the capacity
>>>     of the
>>>     >>> current Droplet CPU and RAM. To upgrade to a system with 2X
>>>     CPU/RAM
>>>     >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
>>>     $96-112/mo.
>>>     >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>>>     >>> processors. The Premium (latest Xeon) option will provide
>>>     enhanced NVMe
>>>     >>> SSDs which would improve DB performance (a bit). I think it
>>>     would be
>>>     >>> worth $112/mo for maximal impact on the responsiveness of the
>>>     forum. AMD
>>>     >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>>>     >>>
>>>     >>> Are there any objections / reasons why we shouldn't go ahead
>>>     and do the
>>>     >>> upgrade. There are a couple of methods available however, the
>>>     down-time
>>>     >>> will probably be 1-3hrs. if no problems. This isn't the first
>>>     system
>>>     >>> upgrade we have done, so I expect no issues.
>>>     >>>
>>>     >>> If any of you wish to see some of the forum performance stats,
>>>     I can
>>>     >>> post another message with numbers and personal observations. A
>>>     quick
>>>     >>> overview of what our daily forum activity looks like:
>>>     >>>
>>>     >>> * Serving 120-160K page views. (about 25% attributed to
>>>     crawlers - a
>>>     >>> different discussion)
>>>     >>> * 6-8K mail messages generated
>>>     >>> * ~1100 User visits (signed-in)
>>>     >>> * ~200 engaged users (liked or posted)
>>>     >>> * ~500 new posts
>>>     >>>
>>>     >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>>>     >>> Droplet use)
>>>     >>> Proposed Droplet upgrade will add 3TB to our total.
>>>     >>> I don't have per-Droplet stats, but last month we transmitted
>>>     ~29TB of
>>>     >>> our 25TB allowance. A $40 overage cost.
>>>     >> Hi Ted,
>>>     >> I am for upgrading the droplet as forum is probably the main
>>>     point of
>>>     >> interaction
>>>     >> in the OpenWrt community.
>>>     >>
>>>     >> But, like others pointed out, we maybe should look for a more
>>>     >> affordable host
>>>     >> then DO as they are quite expensive.
>>>     >
>>>     > We use DO because they sponsor us USD 5000 to use their
>>>     services. If you
>>>     > look at the current costs, upgrading another droplet will cause
>>>     us to
>>>     > use up the USD 5000 before they expire. Please keep that in mind
>>>     before
>>>     > doing anything that will increase cost.
>>>     >
>>>     > Stijn
>>>
>>>     Hi,
>>>
>>>     I think it is fine if we invest some money into the server for the
>>>     forum. From my point of view it is fine to upgrade the server from
>>>     4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
>>>
>>>     We should consider to move some resources to other providers like
>>>     Hetzner, but we can do this later.
>>>
>>>
>>>     Hauke
>>>
>>>     _______________________________________________
>>>     openwrt-adm mailing list
>>>openwrt-adm at lists.openwrt.org
>>>https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>>>
>>>
>>>  Hi all,
>>>
>>>  Is there any interest in my proposal for hosting the forum at Citymesh? (Free of charge)
>>>  Datacenter team has given a Green Light internally.
>>>
>>>  Thanks,
>>>
>>>  Koen (Mobile)
>>>
>>>
>>  Hi Koen,
>>
>>  While I appreciate the offer, I am hesitant to accept it. I remember one of the main driving factors of the LEDE fork all too well. One of them was the downtime of key OpenWrt infrastructure, with the inability of all but one OpenWrt team members to solve said downtime, due to the rest not having access. This is something we absolutely have to avoid from happening again in the future. That said, I would be open to accepting the offer, if the following conditions are met:
>>
>>  At least 3 (preferably 5) OpenWrt team members who are currently involved in maintaining the OpenWrt infrastructure should have full access. As to full access, this includes the ability to call someone at any given time of day to recover something that is broken.
>>
>>  Keeping in mind that cyber attacks are currently happening everywhere, phishing-proof 2FA must be mandatory. In my opinion, the only standard that provides this, is FIDO2 hardware keys.
>>
>>  A signed statement from a major stakeholder at Citymesh, that if a decision is made to retract the offer, we are guaranteed to have at least six months to migrate everything elsewhere.
>>
>>  I know I am harsh on this, but we should learn from the past.
>
>I did not follow the whole thread but when it comes to having a more powerful machine at DO, I could migrate away the ASU server to another provider. It?s less critical infrastructure and rather costly due to CPU and storage requirements.
>
>It?s rather trivial to setup so I could move it within days. Please let me know if that would be of help.
>
>Best,
>Paul

Hi Paul, Koen -

If you guys want set something up to move ASU from DO that would be 
great. Besides cost, it would probably help a bit with our bandwidth 
exfil quota.

Another contributor to high bandwidth usage is our main site wiki. This 
is mostly due to its being hammered by crawlers as was our forum until 
recent changes. I will be working on some blocking and rate-limiting 
functionality on the wiki in the next week or so (I don't think the 
current blocker is working) -- I'm on vacation next week and will start 
on it early in Sept. as well as planning to upgrade the forum system.

/ted




From Jan.Mueller at digitecgalaxus.ch  Thu Aug 29 07:19:35 2024
From: Jan.Mueller at digitecgalaxus.ch (=?iso-8859-1?Q?Jan_M=FCller?=)
Date: Thu, 29 Aug 2024 14:19:35 +0000
Subject: Financial / Infrastructure support of the openWRT project
Message-ID: <GVAP278MB04395435C2810C8AA0B24D45FA962@GVAP278MB0439.CHEP278.PROD.OUTLOOK.COM>

Hey *

My Name is Jan and I am the Product Owner of the team which takes care of sites-connectivity at DigitecGalaxus. To interconnect our sites, we are heavily leveraging openWRT and are very happy so far. Therefore we would like to support/contribute to the openWRT project. We would like to discover how this is done best; we can either provide some one-time or re-occuring donation or provide access to some hardware (probably Hetzner server(s)), where we would take part of covering the costs. 

So is there currently a need for new/more hardware or are donations a more flexible/efficient way to cover your overall costs?

Kind regards 
Jan

From stijn at linux-ipv6.be  Fri Aug 30 12:01:27 2024
From: stijn at linux-ipv6.be (Stijn Tintel)
Date: Fri, 30 Aug 2024 22:01:27 +0300
Subject: Forum host upgrade - proposal
In-Reply-To: <CAPh3n81VHCKbuHzviJANUdaTdBfSnErqT=ijGZhdOPq_5OGkCg@mail.gmail.com>
References: <6693add5-82d0-40e1-9a5b-4a4903fe6122@linux-ipv6.be>
 <9D22D049-2697-4491-81C3-720149ABC387@aparcar.org>
 <CAPh3n82RJNWAX1y0m3Sd3uHRhhp62RNZFYQXkhEZLffRwXbB0A@mail.gmail.com>
 <CAPh3n81VHCKbuHzviJANUdaTdBfSnErqT=ijGZhdOPq_5OGkCg@mail.gmail.com>
Message-ID: <879b3589-c3c8-44de-964b-92f97c3db126@linux-ipv6.be>

On 20/08/2024 19:03, Koen Vandeputte wrote:
> On Tue, Aug 20, 2024 at 5:48?PM Koen Vandeputte
> <koen.vandeputte at citymesh.com> wrote:
>> On Sat, Aug 17, 2024 at 10:13?AM Paul Spooren <mail at aparcar.org> wrote:
>>> Hi,
>>>
>>>> On 16. Aug 2024, at 21:49, Stijn Tintel <stijn at linux-ipv6.be> wrote:
>>>>
>>>> ?Resend because someone decided to send HTML email which tricked my email client to sending HTML email as well, which resulted in my response being rejected.
>>>>
>>>> Please people, stop sending HTML email, and while at it, stop top-posting.
>>>>
>>>>> On 16/08/2024 21:37, Koen Vandeputte wrote:
>>>>>
>>>>>
>>>>> Op do 15 aug. 2024 01:06 schreef Hauke Mehrtens <hauke at hauke-m.de>:
>>>>>
>>>>>     On 8/12/24 22:11, Stijn Tintel wrote:
>>>>>     >
>>>>>     > On 8/12/24 15:33, Robert Marko wrote:
>>>>>     >> On Sat, 10 Aug 2024 at 17:54, Ted Hess <thess at kitschensync.net>
>>>>>     wrote:
>>>>>     >>> Hi all -
>>>>>     >>>
>>>>>     >>> Those of you who visit the forum have probably noticed its,
>>>>>     sometimes
>>>>>     >>> extreme, sluggishness over the last several months. Since
>>>>>     April, I have
>>>>>     >>> worked on and applied a number of attempts at tuning, cleanup and
>>>>>     >>> storage re-arrangement to try and alleviate these issues. As
>>>>>     best as I
>>>>>     >>> can determine, we have have just simply exceeded the capacity
>>>>>     of the
>>>>>     >>> current Droplet CPU and RAM. To upgrade to a system with 2X
>>>>>     CPU/RAM
>>>>>     >>> (4vCPU/8GB -> 8vCPU/16GB) will increase cost from $48/mo to
>>>>>     $96-112/mo.
>>>>>     >>> The difference in cost is a "Regular Intel" vs "Premium Intel"
>>>>>     >>> processors. The Premium (latest Xeon) option will provide
>>>>>     enhanced NVMe
>>>>>     >>> SSDs which would improve DB performance (a bit). I think it
>>>>>     would be
>>>>>     >>> worth $112/mo for maximal impact on the responsiveness of the
>>>>>     forum. AMD
>>>>>     >>> vs Intel - cost is the same. AMD may be slightly higher GHz.
>>>>>     >>>
>>>>>     >>> Are there any objections / reasons why we shouldn't go ahead
>>>>>     and do the
>>>>>     >>> upgrade. There are a couple of methods available however, the
>>>>>     down-time
>>>>>     >>> will probably be 1-3hrs. if no problems. This isn't the first
>>>>>     system
>>>>>     >>> upgrade we have done, so I expect no issues.
>>>>>     >>>
>>>>>     >>> If any of you wish to see some of the forum performance stats,
>>>>>     I can
>>>>>     >>> post another message with numbers and personal observations. A
>>>>>     quick
>>>>>     >>> overview of what our daily forum activity looks like:
>>>>>     >>>
>>>>>     >>> * Serving 120-160K page views. (about 25% attributed to
>>>>>     crawlers - a
>>>>>     >>> different discussion)
>>>>>     >>> * 6-8K mail messages generated
>>>>>     >>> * ~1100 User visits (signed-in)
>>>>>     >>> * ~200 engaged users (liked or posted)
>>>>>     >>> * ~500 new posts
>>>>>     >>>
>>>>>     >>> Footnote on bandwidth usage... (DO charges bandwidth on aggregate
>>>>>     >>> Droplet use)
>>>>>     >>> Proposed Droplet upgrade will add 3TB to our total.
>>>>>     >>> I don't have per-Droplet stats, but last month we transmitted
>>>>>     ~29TB of
>>>>>     >>> our 25TB allowance. A $40 overage cost.
>>>>>     >> Hi Ted,
>>>>>     >> I am for upgrading the droplet as forum is probably the main
>>>>>     point of
>>>>>     >> interaction
>>>>>     >> in the OpenWrt community.
>>>>>     >>
>>>>>     >> But, like others pointed out, we maybe should look for a more
>>>>>     >> affordable host
>>>>>     >> then DO as they are quite expensive.
>>>>>     >
>>>>>     > We use DO because they sponsor us USD 5000 to use their
>>>>>     services. If you
>>>>>     > look at the current costs, upgrading another droplet will cause
>>>>>     us to
>>>>>     > use up the USD 5000 before they expire. Please keep that in mind
>>>>>     before
>>>>>     > doing anything that will increase cost.
>>>>>     >
>>>>>     > Stijn
>>>>>
>>>>>     Hi,
>>>>>
>>>>>     I think it is fine if we invest some money into the server for the
>>>>>     forum. From my point of view it is fine to upgrade the server from
>>>>>     4vCPU/8GB -> 8vCPU/16GB for an additional ~70$ per month.
>>>>>
>>>>>     We should consider to move some resources to other providers like
>>>>>     Hetzner, but we can do this later.
>>>>>
>>>>>
>>>>>     Hauke
>>>>>
>>>>>     _______________________________________________
>>>>>     openwrt-adm mailing list
>>>>>     openwrt-adm at lists.openwrt.org
>>>>>     https://lists.openwrt.org/mailman/listinfo/openwrt-adm
>>>>>
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Is there any interest in my proposal for hosting the forum at Citymesh? (Free of charge)
>>>>> Datacenter team has given a Green Light internally.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Koen (Mobile)
>>>>>
>>>>>
>>>> Hi Koen,
>>>>
>>>> While I appreciate the offer, I am hesitant to accept it. I remember one of the main driving factors of the LEDE fork all too well. One of them was the downtime of key OpenWrt infrastructure, with the inability of all but one OpenWrt team members to solve said downtime, due to the rest not having access. This is something we absolutely have to avoid from happening again in the future. That said, I would be open to accepting the offer, if the following conditions are met:
>>>>
>>>> At least 3 (preferably 5) OpenWrt team members who are currently involved in maintaining the OpenWrt infrastructure should have full access. As to full access, this includes the ability to call someone at any given time of day to recover something that is broken.
>>>>
>>>> Keeping in mind that cyber attacks are currently happening everywhere, phishing-proof 2FA must be mandatory. In my opinion, the only standard that provides this, is FIDO2 hardware keys.
>>>>
>>>> A signed statement from a major stakeholder at Citymesh, that if a decision is made to retract the offer, we are guaranteed to have at least six months to migrate everything elsewhere.
>>>>
>>>> I know I am harsh on this, but we should learn from the past.
>>> I did not follow the whole thread but when it comes to having a more powerful machine at DO, I could migrate away the ASU server to another provider. It?s less critical infrastructure and rather costly due to CPU and storage requirements.
>>>
>>> It?s rather trivial to setup so I could move it within days. Please let me know if that would be of help.
>>>
>>> Best,
>>> Paul
>> Hi Stijn & all,
>>
>> I totally understand your arguments and fully respect the requirements.
>>
>> I've forwarded this internally to the right people to check your requirements
>> and will come back to you on that.
>>
>> If this would be rejected for some reason, maybe we can aid in
>> different needs which are less mission critical in the future. :-)
>>
>> Thanks,
>>
>> Koen
>
> Hi all,
>
> And I already got a reply back internally ..
>
> Please clearly state following requirements:
>
> - Network: estimates of current throughput + monthly volume
> - Disk space currently required
> - CPU (speed + cores)
> - RAM (amount)
>
> Stijn,
>
> Please provide more details about this part:
> - Keeping in mind that cyber attacks are currently happening
> everywhere, phishing-proof 2FA must be mandatory. In my opinion, the
> only standard that provides this, is FIDO2 hardware keys.

Hi Koen,

I was mostly thinking about what we currently have at DO. We currently 
have 9 admins there all with 2FA enabled, but this is to access the 
control panel. Might not be relevant if your company doesn't have such a 
control panel to deploy e.g. VM instances. It's also not using phishing 
resistant 2FA it seems, as it appears I'm using TOTP there.

What I would recommend is that everyone of the team gets at least 2 FIDO 
keys and we start using FIDO2 backed SSH keys [1] on all our servers, 
but that's not directly related to this discussion.

>
> Your other requirements are fine and reasonable.
> Communication would also happen with the DC team and/or support
> directly, not through me ofc to avoid single point of failure.
Excellent!
>
>
> If the resource specifications are accepted, I would propose to loop
> some of you in with the DC team for further discussion if that's ok.


Thanks,
Stijn


[1] https://developers.yubico.com/SSH/Securing_SSH_with_FIDO2.html



