[LEDE-DEV] Release 17.01.0 binary packages have changed and SDK inconsistency

[Alberto Bursi]

On 03/29/2017 05:39 PM, James Feeney wrote:
> Hi Alberto
>
>> Also the kernel is handled like a package for the build system, but
>> since most devices expect it outside of root filesystem in various ways,
>> it's added to the firmware image the way the device's bootloader expects it.
> I haven't seen a preferred term for the class of this combined.  LEDE does name each of them individually, based upon the target
> architecture.  It would be nice to have some generally recognized single name
> for this class, "the kernel-fs packages" or some such.
>
> Of course it's a little complicated, since most traditional Linux distros target
> only one or two hardware architectures, and LEDE targets many many different
> target architectures.  But then, that's the whole point of LEDE.
>

I've seen it called "image", as it is basically a disk image ready to be 
flashed on raw flash (for most devices), or simply "firmware" as the end 
result is a complete and bootable firmware if installed correctly.

> So, if I understand, there always exists a set of hashes to verify that, at any
> time in the future, a package can be built which is identical to any package
> built in the past.

The hash is to verify integrity of source package downloaded from 
upstream, afaik there is a hash after compilation, but that is generated 
automatically and listed in the package lists read by opkg only. It's 
not stored permanently.

You can revert to an older version of LEDE git and compile the same 
package with its build system, assuming you still have access to the 
same source package or git repo or whatever that was used before.

> Bastian also mentions "checkout LEDE and all feeds at a specific timestamp" as
> an approach.  This suggest two general tracking mechanisms: 1) tracking by hash,
> and 2) tracking by time-stamp.  Is one of these mechanisms, or are both of these
> mechanisms, used universally in the LEDE package management system?

I think he meant that they are doing like that to make their own 
downstream LEDE-based firmwares. They tag a timestamp in LEDE snapshot 
or whatever, and stick to it until they decide to tag the next.

In any case LEDE is either downloading a source archive from a specific 
release version (and using the hash) or fetching source from a specific 
commit in a versioning system like git or subversion or whatever.

>
> Hmm - it is not entirely clear to me that, if the patch set changes in the
> future, the version number of the package will be distinctly incremented, as
> would be pretty standard for any other Linux distro.  Does each package have a
> minor version number, in the source and binary repositories, to allow
> distinguishing patch set changes, used in each build of every package?
>

There is a secondary version number, yes.

Packages have PKG_VERSION that shows the upstream version and 
PKG_RELEASE that is used to show changes on the LEDE side. If you see a 
package that lists version as 123-1, it's major version is 123 and its 
minor version is 1.

Packages with a timestamp use that instead of major version, and they 
still have a minor version.

for example  2016-09-21-42ad5367-1 where the last "1" is the minor version.

You can see all versions used in the first page of the table of packages 
in the wiki https://lede-project.org/packages/start

Or by browsing a Package.manifest in the package download folders 
https://downloads.lede-project.org/releases/17.01.0/packages/aarch64_armv8-a/base/Packages.manifest

> Perhaps, assuming that these distinct version numbers exist, and that these
> historic time-stamp or hash verified sources are archived somewhere and available.

The hashes and time stamps of source archives or commits are written in 
the package's makefile in the git repository (either main repo or 
community package feeds), you can see full history of changes to that 
file with git.

There is a cache server that stores source archives because it's more 
convenient than having the build bots spamming upstream download servers 
and it is a convenient fallback if the upstream download disappears, it 
should keep stuff for a long while (as long as the release is supported 
for sure, as it is used by build bots).

it is here http://sources.lede-project.org/

Also OpenWRT's source cache server is used (as a fallback maybe) 
https://downloads.openwrt.org/sources/

> It sounds like the LEDE package management system has changed and improved over
> the past months.  It would be great to see your description of the LEDE package
> management system and build system on the wiki!

Ok, will move this to wiki in a few days.

-Alberto