[RFC] A change to the way packages are built

[Daniel Dickinson]

On 16-05-04 08:38 PM, David Lang wrote:
> On Wed, 4 May 2016, Daniel Dickinson wrote:
> 
>> Hi,
>>
>> I wanted to know if there is any appetite for an idea I've been kicking
>> around in my head for a while for package builds.
>>
>> Basically one builds a minimal SDK and does kind of like Debian where a
>> git commit to a package kicks of a build of pristine environment which
>> builds only the package and it's dependencies.
>>
>> The goals are:
>>
>> 1) Avoid the cost of monolithic rebuilds of every package in the enabled
>> feeds for every snapshot (obviously packages depend on specific kernel
>> versions do have to be rebuilt when the kernel is rebuilt).
>>
>> 2) Faster turnaround on package builds when a package change is
>> committed.
>>
>> 3) Decouple (where possible) packages from the core.

This part is probably the hardest...

>>
>> Thoughts?
> 
> The problem I expect you to run into is dependencies between packages.

Yes, although I primarily thinking of it from the package failed because
a package it depends on fails or has default configuration that doesn't
work with the package. (i.e. the reverse of what you mention).

> You really want to have multiple stages
> 
> 1. compile the package in isolation (avoid any "won't build" or "won't
> pass self-test" situations)

This is definitely one of the major reasons I want to do this - instead
of having huge logs from the buildbots to wade through to find problem
(plus having to build *everything* to get the buildbot logs), a subset
if easily found errors, which could show up quickly (i.e. kind of like
launchpad's PPA system where you submit a package to build and pretty
quickly see if there is an issue; although IMO if builds fail
egregiously (i.e. committer/patcher would have caught it if they had
actually built on at least one arch) then the committer would have
explain why and if it happens 'too often' then potential suspension of
commit privileges); it should be a substitute for testing, more to catch
archs or configuration options the patcher didn't realize needed to be
tried).

> 
> 2. comile all the packages that depend on this package, see if their
> self-tests still work.

Self-test part only works if packages have self-tests and you're
building on the same arch as the pacakge is for, but at least verifying
the the dependant packages compile would be a good step.

> 
> 3. compile and test everything together to catch conflicts between
> package A and package B both making changes that end up conflicting when
> building package C.

Ok, I can see that this is a case for at least have full builds of
everything even if not as frequently as without this.

> 
> automated tests for #1 would be a good start.

Agreed (although not so much the self-tests since mostly we're
cross-compiling and actually running the code is not normally an option).

Regards,

Daniel