[LEDE-DEV] Better backups to make upgrade easier

[Luiz Angelo Daros de Luca]

Hello,

Upgrade process with LEDE/OpenWrt installations aren't as smooth as a
user would expect. I know that PC have a lot more resources but their
upgrade process is normally very straightforward. They can upgrade
thousands of packages with little to no intervention.

On LEDE, it is normally not feasible to simply backup all overlay for
a upgrade. It works for a disaster recovery but, in an upgrade, extra
installed packages might break with the new runtime env.

The standard way to upgrade and keep configuration is using the system
generated config backup. It works with simple installation but they
will fail whenever you need any extra package. It is also designed to
ignore all files but the ones listed as config files or in
sysupgrade.conf. So, a user must doublecheck the list of files in
backup before every upgrade in order to not miss an extra script file,
a ssh key, etc.

At the same time, backup saves unneeded stuff. For example, every
backup includes /etc/profile, /etc/protocol. Do we really need all
those files, even when they are still untouched? opkg already keeps
hash of all of them for "opkg list-changed-conffiles". Someone that
upgrade from CC simply misses all changes to those file, which
disables, i.e., the use of /etc/profile.d/. (BTW, /etc/profile.d/
files are not listed as conf file for backup). Who knows what else
will break in the future.

The user must take note on every extra package was installed on the
system before an upgrade in order to know what is needed to be
reinstalled post upgrade. There is not an easy way to list every
package I asked to be installed. On a overlay system, I can deduce it
by looking at /overlay/upper/usr/lib/opkg/info/*.control.
/usr/lib/opkg/status was my first try but it does not keep the
installation order and new packages appear in the middle of the list.
It is also interesting to ignore those packages installed as
dependency (Auto-Installed: yes at /usr/lib/opkg/status; not shown by
opkg).

In my upgrades, I created a new type of backup composed of two parts:
1) save the list of packages I asked to be installed (not
auto-installed, not built-in)
2) save all files in overlay, excluding those from packages but
including changed config files.

This is not ideal, as it does not track changed non-config files from
packages (should it? maybe warn about it), but it works flawlessly
with 99% of my setups and saves me a lot of time.

With those two files, I can flash a new version already providing the
file 2) with the option "-f". After the installation, I simply batch
install all packages from 1). If any package from 1) is needed to
reach the target system, I create a custom image adding those files
and flash it, again with the 2) config file.

This is my backup script for reference
https://drive.google.com/open?id=0B54xzz44RpW6dk1WRTFXX1kwdU0
1) is backup-*-installed_packages_by_user.txt and 2)
backup-*-overlay_diff.tgz. It also generates extra backups options.

Would it be interesting to:
a) stop saving not changed config files in current backup (this is
already breaking things)
b) create an "expanded backup", based on 2). Add it to luci.
c) make luci accept a config file during the flash process (as sysupgrade does)
d) save the list of installed packages (all, after flashed, user
installed or all of them) in a different file or inside every backup
e) there is already a GSoC for the installation of the extra packages
(required for unattended upgrade). However, it would also be possible
to inform during the flashing that the system must try to download and
install those extra packages (included in the backup file) as soon as
internet is up. This does not need to work for all scenarios as it is
not an unattended nor automatic upgrade.

I can create those patches but, at first, I would like to discuss if
the idea would be accepted.

Regards,

---
     Luiz Angelo Daros de Luca
            luizluca at gmail.com