[LEDE-DEV] Release 17.01.0 binary packages have changed and SDK inconsistency

Wed Mar 29 17:45:51 PDT 2017

On 29/03/17 20:57, Jo-Philipp Wich wrote:
> Hi Pau,
> 
> I will not reiterate the arguments made elsewhere in this thread but
> merely describe why the version tagged 17.01.0 SDK uses fixed feed sources.
> 
> The main objective here was to make the build reproducible in a way that
> you can use the 17.01.0 SDK to build artifacts resembling the released
> 17.01.0 images. Note that I specifically mean images here and not the
> package feed universe. No effort was made to freeze binary package
> repositories on the server.
> 
> Only a tiny subset of all binary packages is frozen and tied to a
> specific release, this subset consists of a) source packages marked as
> "nonshared" in the source trees and b) of kmod packages.
> 
> All other packages are rebuilt incrementally from the latest HEAD of the
> "lede-17.01" branch in each feed repository.
> 
> The reasoning behind rebuilding non-"nonshared" and non-kmod packages is:
> 
>  - by definition any commit made to a release maintenance branch is
>    either a bug fix or important update (e.g. to address security issue)
> 
>  - it is expected that commits made to release maintenance branches
>    are backwards, abi and config compatible (e.g. no libfoo 1.x -> 2.x
>    bump)
> 
>  - it rarely (if ever) makes sense to stick to a known old version of a
>    binary package, at least when assuming that a newer version fulfills
>    the compatibility requirements raised above
> 
>  - the vast majority of such updated packages is not bundled with the
>    image root filesystem so an "opkg install" of a 3rd party package
>    will fetch the latest release maintenance branch version of it, not
>    some outdated version that happened to be current at the release
> 
> This of course leads to a number of considerations and constraints one
> has to be aware of:
> 
>  - there needs to be a certain "branch discipline" which means that
>    commits introducing incompatible ABI changes or other disruptive
>    updates are to be avoided
> 
>  - critical updates (such as CVE fixes) within the release maintenance
>    branches made to packages which are part of the released images need
>    additional care; while updated/fixed binary packages will be
>    available for users to install via "opkg upgrade", there still is a
>    need to roll a new point release with updated images containing the
>    newer versions of such builtin binary packages from the get-go since
>    squashfs based root file systems mostly make in-place updates
>    impossible due to inherent space constraints.
> 
>  - Without local modifications to the feed manifest, it is indeed not
>    possible to use the released SDK as-is to build 100% identical
>    packages for a given release at a later point in time
> 
> 
> The decision to release point version SDKs with fixed feed URL pointers
> is not set in stone and can be revised; it was merely done to address
> the expressed need for being able to exactly rebuild a release image as
> found on the download server. It also serves as a reference to what
> components in which version were used to form a distribution release at
> a specific point in time.
> 
> The buildfarm incrementally producing updated binary repositories for
> the current release is not actually utilizing the v17.01.0 SDK but the
> one found at
> https://downloads.lede-project.org/releases/17.01-SNAPSHOT/targets/*/*/lede-sdk-17.01-SNAPSHOT-*.tar.xz
> which always represents the state of the latest release maintenance work.
> 
> 
> To summarize; the 17.01.0 SDK is good for:
> 
>  - building out-of-tree kmods that fit to the kernel found in released
>    images
> 
>  - identically rebuilding release images with the same set or a subset
>    of the images included in official images
> 
> It is (out of the box) not suitable to:
> 
>  - build the latest packages available for a release as found on the
>    binary download server
> 
> 
> In the future it might be possible to have both a copy of the original
> repository state at the time of the release and a kind of "updates"
> repository which is overlaid over the release repository and only
> containing binary packages which have been updated since the release.
> 
> We are, however, not there yet as both buildroot and opkg infrastructure
> is lacking to support such an overlay repository scenario which is why I
> opted for preferring current binary packages over frozen repositories.
> 
> What we can do on a short notice (e.g. for 17.01.1) is retaining a copy
> of the original initial repository but build images to use another
> repository which is continuously getting updated.
> 

Hi Jow.

Thanks for taking the time to clarify. I think I just understood wrong
the concept of release for LEDE. All your points sound reasonable to me.
The only minor thing to comment is that instead of using static commit
hashes into feeds.conf.default IMO would be better to use TAGS to easier
identify where the release is pointing when looking at the feed Git history.

Cheers.

> Now on to your specific questions below:
> 
>> I am using the SDK and IB from LEDE 17.01.0 release (mips_24kc).
>> I've been using it successfully the last days until now. I did not
>> change anything but I get the error:
>>
>> * opkg_install_pkg: Package luci-lib-nixio sha256sum mismatch.
>> Either the opkg or the package index are corrupt. Try 'opkg update'.
>>
>> Then I went to downloads.lede-project.org [1] and I see that there
>> are new compiled packages with date "Tue Mar 28 18:35:15 2017". 
>> Luci-lib-nixio is one of them [2] (which now is published on the 
>> repository with a new version).
>>
>> In the other hand the feeds.conf.default file included in the
>> release SDK points to the specific commit 
>> a100738163585ae1edc24d832ca9bef1f34beef0 from  Sat Jan 28 01:38:06
>> 2017.
> 
> Correct.
> 
>> The SDK published then is not consistent with the current package
>> binary repositories.
> 
> Right, or more specifically it is only consistent with the
> https://downloads.lede-project.org/releases/17.01.0/targets/*/*/packages/
> portion
> of packages, which is the main scope of the release SDK.
> 
>> So my question here is: what's the point for publishing a Release if
>> the packages included are changing?
> 
> The main point of a release has always been to provide a fixed set of
> images with a fixed set of included core packages and kmod-packages that
> remain installable long after the image has been flashed.
> 
> The current scope of the release is not to provide one frozen copy of
> all optional 3rd party packages which stays as-is forever but instead to
> provide a less-volatile more carefully maintained binary package
> repository which exclusively receives bug fixes, security fixes and
> assorted backports, like you would expect when running a stable desktop
> Linux distribution.
> 
>> Is this the expected behavior?
> 
> Within the scope explained above it is expected behavior.
> 
> If you intend to produce self built packages which are always
> bit-identical to the current ones on the downloads repository; use the
> 17.01-SNAPSHOT SDK or alter the 17.01.0 SDK feeds.conf to exchange the
> fixed commit pointers with ";lede-17.01" branch designations (which is
> essentially the same as using the 17.01-SNAPSHOT SDK minus possible
> buildroot/toolchain fixes).
> 
> 
> I hope the mail cleared things up a bit. I know that some of this might
> appear unusual, confusing or complex to casual outside visitors but most
> constraints we're struggling with here are either rooted in opkg
> deficiencies or the non-writable-root-squashfs property of many target
> architectures. What we're trying to do here is maintaining a balance
> between security (or currentness in general), reproducibility,
> convenience and resources.
> 
> 
> Cheers,
> Jo
> 
> 
> 
> _______________________________________________
> Lede-dev mailing list
> Lede-dev at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/lede-dev
> 

-- 
./p4u

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/lede-dev/attachments/20170330/81c1916c/attachment.sig>