[LEDE-DEV] Release Preparation Questions

[David Lang]

replying to details other than the feeds concept/

On Sun, 20 Nov 2016, James Feeney wrote:

> Ok - thanks for taking the time to explain this versioning process.  So now, I
> should take the time to "feed it back" and see if I understand.  Ha!
>
>> it does not require twenty different variables but it has to deal
>> with (currently) five independent repositories:
>> - feeds.conf.default: git://git.lede-project.org/feed/packages.git
>
> Hmm - try Google again:
>
> https://git.lede-project.org/

the git:// URL format isn't something you can put in a browser, it's the URL you 
can provide to git to most efficiently fetch (and otherwise interact) with the 
repository. Most repositories are also available via http:// and https:// at the 
same domain, but it's more efficient on both the server and client to use the 
git:// URL when cloning a repository.

> Ah ha.
>
> feed/packages.git	Mirror of packages feed		Git	119 min ago
> feed/routing.git	Mirror of routing feed		Git	2 weeks ago
> feed/telephony.git	Mirror of telephony feed	Git	4 months ago
> ...
> project/luci.git	Lua Configuration Interface...	Git	28 hours ago
>
> Ok.  I don't know what I'm seeing yet, but ok.

these are different repositories hosted at that URL

>> ... described in feeds.conf.default within the base repo.
>
> Hmm - still drawing a blank...

There is a file in the LEDE/OpenWRT repository that says what feeds it knows 
about

> Now, revisiting the concept of "version number":
>
> 1) "base" repository - https://git.lede-project.org/?p=source.git
> 2) "feeds" - described in feeds.conf.default within the base repo.
>   1 src-git packages https://git.lede-project.org/feed/packages.git
>   2 src-git luci https://git.lede-project.org/project/luci.git
>   3 src-git routing https://git.lede-project.org/feed/routing.git
>   4 src-git telephony https://git.lede-project.org/feed/telephony.git
> 3) release branch number - an input variable, major.minor version, usually
> year.month - or instead, the number of the build being produced from a branch.
> There were two different definitions offered.  I'm guessing that you meant the
> "build number" after the "major.minor" version number.

not quite

the branch would be Year.Month, tags/builds within that branch would be 
Year.Month.Build

so if a branch is made in November 2016, it would be the 16.11 branch, and 
everything along it would be 16.11.X

This way if there is a fix added in July 2017, it would still be 16.11.X with a 
higher X

This means that you can have a 16.11.x that's got some newer code in it thatn 
17.4.y, but there will be a lot of updates in any 17.5 that are not in anby 
16.11

> 4) git tags - exactly match the source state; "commit hash".

git tags are a human-readable shortcut to refer to a particular snapshot of a 
git repositry, but pointing at the unique hash that referrs to that state.

> 5) builds - the binary artifacts produced, branch release builds.
> 6) master - master branch of the "base" repository.
> 7) release branch - used for *all* builds within a major.minor version; "source
> branch".
> 8) subrelease - Any release within a base version; patch levels; security updates.
> 9) nickname - Symbolic name given to an entire release branch; all 16.11.*
> releases will carry the same nickname.

yes.

> Hmm - it is still not clear to me how the term "branch" will apply here,
> because: 1) the hierarchy of "base.feeds" is referencing four *other* git
> repositories, each of which has its *own* "head" branch name, each of which - I
> assume - may be changed or selected independently; and 2) the "feeds" link list
> does *not* have any version information within it, itself.

The branch is really based on the LEDE maintained git repository, ignoring the 
feeds that are not maintained by LEDE or OpenWRT.

There is an interesting question of how to refer to what state of each feed you 
use for a release. Currently OpenWRT doesn't even try to do this. The feed 
version you get is the feed version that exists at the time you last pulled from 
the feeds.

This hasn't been a big problem in practice, but it does mean that you don't 
really have a repeatable build. See my other e-mail about submodules for a 
discussion on this.

> It would be possible, I suppose, that each of these linked repositories could be
> *required* to use the *same* versioned naming scheme.

can't be done, these other repositories are not under our control. The most we 
can do is reference exactly which release we use..

>> If a builtin package or the kernel itself gain security fixes,
>> the images need to get rebuilt since people cannot update those
>> components with opkg.
>
> This is what is so unusual for me about LEDE/OpenWRT.  It is not a simple
> program package, where every component is versioned in lock-step.  But neither
> is it a traditional distribution, in which all of the components can be
> versioned and installed independently, especially as in a rolling distribution.
> And it is not a "glacial" distribution, like Debian, where the versioning
> measures time in years.  Perhaps it is more similar to Cyanogenmod, which has
> dated "nightly" and "snapshot" builds.  The term "release" is inappropriate in a
> fast moving environment like that.  The term "build" seems better suited.

actually, the LEDE/OpenWRT releases are long, they are trying to get them down 
to only a year or so.

What most people end up using is the roughly the equivalent of the nightly 
builds, but even more granular in that people grab the source code and compile 
it, so they get up-to-the-minute changes, not even nightly snapshots.

> Cyanogenmod uses the term "manifest" instead of "feed", and uses "repo", a
> specialized "repository management tool that we built on top of Git."  You are
> familiar?  In a sense, the top level of the build system is the "manifest" file
> "default.xml" with its own web page, https://github.com/CyanogenMod/android.
> Hmm - LEDE has a program "[source.git] / scripts / feeds", which serves the same
> function as "repo", yes?  Maybe I begin to understand.  And then, the term
> "feed" would actually mean one of several git source code repositories
> enumerated in the LEDE "manifest".  And "feeds" means a LEDE git repository
> management and build utility.

more or less. OpenWRT was stuck for a long time using subversion instead of git, 
so things developed differently, but you seem to be mapping the concepts more or 
less correctly.

> I think that, essentially, I must understand how a LEDE build version number can
> be related to an instantaneous version of four independent git repositories.

ignore the feeds for now, they are not part of the LEDE release.

> Would it be useful to version the manifest, and have the versioned manifest
> enumerate specific versions of the manifest components?  I assume that each
> component could specify an instantaneous snapshot version of each master branch
> of that component?  Perhaps git tags could be included inside the manifest, and
> no one would have to memorize or recognize or refer to these, because the
> manifest itself would have a human readable version number, as in
> "year-month-build".

that would be inventing another layer of abstraction, I don't think we need to 
do that. But this is the submodule discussion I raised in another branch of this 
thread.

>> Did you ever build the complete LEDE or OpenWrt from scratch?
>
> Ha!  NO! That's the issue for me.  The linux kernel - ok, I'm familiar with
> that.  LEDE/OpenWRT - I'm still trying to wrap my head around it!

The first thing to do is to ignore the feeds and all software involved in them..

> The terminology would make more sense to me if "base" referred to a "base
> package" which include the linux kernel and required utilities, unique to each
> hardware system,

it does.

> even though it were a "package" that could not be installed by
> opkg.  As I understand, the kernel and the root file system have to be installed
> together, as a matched set, though I cannot say why.

partly for historical reasons, partly because LEDE/OpenWRT are usually installed 
on systems that have very little storage, so you don't have a 'normal' 
filesystem that you can replace different parts of, you have a compressed 
filesystem where everything you are installing has to be combined into one blob 
and installed all at once.

Thus, while a normal linux distro creates a filesystem on the device, they loads 
the kernel and a bunch of other packages on there, and the build process creates 
a bunch of individual packages, the LEDE/OpenWRT build process takes a selection 
of the packages that are compiled and combines them into a compressed image 
(frequently even including the bootloader) that can be burned directly into the 
flash of the device.

>  The "version number" would
> imply the version of this LEDE "base package".  The LEDE "source" would refer to
> just the git repository "https://git.lede-project.org/", and the git directory
> "source.git" would instead be named "build.git". The file "feeds.conf.default"
> could be called "manifest.default", which would be a concept shared with
> Cyanogenmod.

I hate to disappoint you, but Cyanogenmod's terminology is pretty unique to it, 
it's not universal. It makes more sense to you because you ran across it first, 
but changing the terminology in LEDE would just add to the confusion of the 
exisiting LEDE/OpenWRT users and introduce incompatibilities with years of 
documentation and how-tos

The version does refer to the exact state of the LEDE maintained git repository 
(feeds are a different/complicating issue)

David Lang