OSUOSL datacenter migration and hosting costs for buildbot workers

[Baptiste Jonglez]

Hi Pono, all,

As part of the OSUOSL datacenter migration, we have received an offer from
them to upgrade our build workers with much newer hardware (servers with
2x Xeon Platinium 8280 CPU, perfect for running a large number of builds
in parallel).  We could benefit from having 2 such servers.

But before accepting, I feel we should clarify the funding aspect.

@Pono: did you have a chance to discuss hosting costs with SFC and OSUOSL
already?  What kind of budget could we allocate to infrastructure costs?

@all: any opinion on paying hosting costs for build workers?  In my
opinion, the limiting resource is sysadmin time (mostly ynezz and me in
our spare time).  OSUOSL is really great in that regards, because their
hosting is very stable and long-term.


As a reminder, we have an overview of infrastructure costs here:

  https://openwrt.org/infrastructure#hosting_infrastructure_and_costs

Please update it in case it's not up-to-date.

Of course we need to keep running build workers at multiple providers
for redundancy (many thanks to the people/org sponsoring build workers!)

Baptiste

On 07-07-25, Daniel Pono Takamori wrote:
> Heya, Pono from Software Freedom Conservancy (OpenWrt's fiscal home)
> here.
> 
> Just wanted to quickly chime in a mention that we at SFC are working
> with the OSL, together with our member projects, to figure out the best
> way forward for hosted infrastructure. Sounds like it's been working out
> with hosting buildbot workers over there, so if that seems to be the
> preferred option, we can work on getting a contract with OSL (and find
> out hosting costs). On that note I wanted to mention that there are
> funds available at SFC for things like this, so we can avoid having
> members pay for CI!
> 
> I'll update again after we get a chance to talk internally and with
> Lance at OSUOSL about the best way forward for SFC and its member
> projects to continue to host over there.
> 
> Thanks!
> -Pono
> Community Manager
> Software Freedom Conservancy
> 
> On Mon, Jul 07, 2025 at 11:21:53PM +0200, Baptiste Jonglez wrote:
> > Hi,
> > 
> > OSUOSL is migrating to a new datacenter that will increase their hosting
> > costs, and they want to discuss how we could possibly contribute to costs.
> > Details quoted below.
> > 
> > They currently provide us 6 build workers (2 physical hosts and 4 VMs):
> > 
> >   https://openwrt.org/infrastructure#buildbot_workers
> > 
> > This infra has been working quite well since 2018, with different
> > iterations.  We haven't paid anything so far.
> > 
> > I see several possible directions:
> > 
> > 1) establish a contract and start paying OSUOSL for the service using OpenWrt funds
> > 
> > 2) stop using these buildbot workers
> > 
> > 3) keep using these buildbot workers for free (same as today)
> > 
> > Personally I would prefer to continue working with OSUOSL, and I think we
> > have the funds to contribute financially.  They have been very
> > professional and responsive, the level of service is the same or even
> > better than companies like Hetzner.  We don't need to spend too much time
> > since they take care of everything (e.g. initial OS installation, hardware
> > repairs)
> > 
> > We could even maybe increase the number of workers: it seems more sustainable
> > than having project members paying buildbot workers from their own pockets.
> > 
> > What do you think?
> > 
> > 
> > OSUOSL request below:
> > 
> > > Due to significant infrastructure issues at the Kerr Datacenter, including a
> > > critical failure in one of the two main UPS systems and the high cost of
> > > necessary repairs, Oregon State University has made the decision to vacate this
> > > facility rather than invest further in it. As a result, OSL is required to
> > > relocate all of its equipment within the next few months (potentially sooner).
> > >
> > > Relocating to a new datacenter will increase OSL's operational costs for power,
> > > space, and connectivity compared to Kerr. Historically, OSL covered most costs,
> > > but higher expenses in new facilities mean sustainable hosting for dedicated
> > > physical hardware requires projects to contribute more directly to covering
> > > these operational expenses.
> > >
> > > For projects with existing contracts:
> > >
> > > We don't anticipate an immediate increase in your contractual costs. However,
> > > higher OSL costs overall may necessitate future discussions about pricing upon
> > > contract review/renewal.
> > >
> > > For projects without contracts:
> > >
> > > Facing significantly higher operational expenses in a new facility compared to
> > > Kerr, sustainable hosting for dedicated physical hardware will require projects
> > > to contribute more directly to covering these operational expenses. We
> > > understand that fully covering the total costs might be challenging for some
> > > projects, and we are flexible in working with you to find a solution. Our aim is
> > > to ensure the costs directly associated with hosting your physical equipment are
> > > addressed to make continued co-location sustainable.
> > >
> > > We want to partner with you to find a workable solution for contributing to
> > > these costs. We offer two primary paths for contribution: establishing a formal
> > > hosting contract (our preferred method) or providing an annual donation based on
> > > an estimate of your hosting costs. We are committed to working with you to find
> > > a mutually agreeable solution and will not remove any co-located hardware until
> > > a solution is in place. Please let us know in your reply if you would like to
> > > discuss these contribution options in more detail.
> > 
> 
> 
> 
> > _______________________________________________
> > openwrt-adm mailing list
> > openwrt-adm at lists.openwrt.org
> > https://lists.openwrt.org/mailman/listinfo/openwrt-adm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20251115/6acdc701/attachment.sig>