[OpenWrt-Devel] [RFD] sysupgrade and configuration management

[Paul Oranje]

Hyjacking on what luizluca at gmail.com wrote on 18 aug. 2018 at 01:49,
with subject [OpenWrt-Devel] [PATCH v3 0/4] base-files: add new backup options

> Today, every file listed in /etc/sysupgrade.conf, /etc/config,
> marked as changed conffile and others will be in backup. Some of
> these files from previous OpenWrt version might break something.
> /etc/profile is a good example of what should not be in backup if
> unchanged. Also, any conffile that has a new required parameter
> might break after restore.
> 
> When the user changes a file, it is expected that he/she knows that
> he/she is doing. The problem is when OpenWrt replaces a file with one
> from a previous version that the user might not even know that it exists.
> 
> The new '-u' option asks backup to (oportunisticly) skip any file that
> is equals to /rom. If a system does not have /rom, it simply does
> nothing. /rom does not need to actually be squashfs. If the user is using
> ext4, he/she could simply copy files into /rom before the first change
> is made. IMHO, it should even be the default behavior.
> 
> A backup might also miss important files for the user. The user must
> insert every single file needed in /etc/sysupgrade.conf in order to get
> it into a backup. However, it is easy to simply miss one. '-c' option
> does try to save everything, but limited to /etc, probably to skip code
> files. Saving /overlay also works but only when restoring to an
> identical OpenWrt version. The new '-o' is equivalent to saving
> /overlay, but it skips any files that came from a package, except those
> marked as a changed config file, sysupgrade.conf or /lib/upgrade/keep.d.
> It does work with '-u', skipping files touched but reverted to the
> original state.
> 
> After the user seeded a new OpenWrt with a backup generated with '-o'
> and '-u', the next natural step is to reinstall all previously installed
> packages. '-k' adds a file into the backup containing the list of
> installed packages and also its origin (rom or overlay). It is a one-line
> script to reinstall them all.
> 
> My normal upgrade procedure is:
> 
> # sysupgrade -o -k -u openwrt-new-version.img
> # <auto reboot>
> # opkg update
> # grep "\toverlay" /etc/backup/installed_packages.txt | cut -f1 | xargs -r opkg install
> # rm /etc/backup/installed_packages.txt
> # reboot
> 
> Those options could be used by Luci, exposed to user during an upgrade.
> The (re)installation step could even become automatic (on demand) or
> offered to the user when Luci detects /etc/backup/installed_packages.txt
> presence.
> 

Very nice, but having read the mail thread around Luiz' his patch, I want to ask for a discussion, but outside of that patch.

AFAICT from the discussion it follows that sysupgrade, opkg and UCI should be kept (as much as possible) orthogonal. From the thread: opkg cannot always be assumed to be available, external config management may need all configuration files, not just those that changed and not just diffs, config files may need changes, etc.

The backup functionality in sysupgrade serves a common case of keeping config files over upgrades where that's needed because the rootfs/overlay is re-created. Cases where sysupgrade does not destroy the (overlay) filesystem contents also exist and obviously config files aren't lost than.

After/during a sysupgrade a normal step is the restoration of the previous configuration, i.e. of config files *and* extra previously installed packages - currently sysupgrade does not handle re-install of extra user installed packages. This functionality could also serve cases where a certain configuration is to be put on a new or other device. Where the filesystem its contents is not lost during sysupgrade, this would just amount to an opkg upgrade of all outdated packages after sysupgrade has upgraded the kernel and possibly the rootfs.

In the more common sysupgrade case though (overlay contents are lost), some means is needed to preserve configuration data in order to be able to restore the configuration. Where this data is preserved during sysupgrade depends completely on the device specific sysupgrade mechanism; *what* data is saved and *how* it is re-applied is preferably opaque to, and independent of, sysupgrade.

Also, during upgrades new versions of packages may require updates to the config files. Preserving customisations of the config files cannot just be handled with back-up/restore and may require applying a diff or user intervention. How to deal with that can really only be determined by the package maintainer and so must be handled by the package itself. Anyhow, this clearly is in the domain of opkg and applies to upgrades outside of sysupgrade as well.

In short, factoring out the configuration management functionality (CM) seems a good idea. The sysupgrade subsystem handles with upgrading kernel and rootfs and may invoke CM functions to preserve the user configuration. CM can could also take care of user installed packages. CM calls opkg to install and/or upgrade additional packages. Truly general back-up (e.g. as a defense against data loss) is outside the scope of sysupgrade.

Quite a few efforts have already been made toward some kind of CM, notably by offering this as an external service. The task to factor out generic CM  functionality that can be called upon in different contexts is probably quite complicated and comments on that idea are welcome.

What would be a practical API, if any, for CM ?
What (minimum) configuration data would be needed by CM, i.e. what's needed besides config files to store user install selections ?
What's needed to enhance opkg so that it can handle in a general way updates to config file while preserving user settings ?
Compatibility issues ?

Paul
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel