[OpenWrt-Devel] [RFD] sysupgrade and configuration management

[Paul Oranje]

Ola, see remarks below, bye,
Paul

> Op 28 aug. 2018, om 07:36 heeft Luiz Angelo Daros de Luca <luizluca at gmail.com> het volgende geschreven:
> 
> Hi Paul,
> 
>> Very nice, but having read the mail thread around Luiz' his patch, I want to ask for a discussion, but outside of that patch.
>> 
>> AFAICT from the discussion it follows that sysupgrade, opkg and UCI should be kept (as much as possible) orthogonal. From the thread: opkg cannot always be assumed to be available, external config management may need all configuration files, not just those that changed and not just diffs, config files may need changes, etc.
>> 
>> The backup functionality in sysupgrade serves a common case of keeping config files over upgrades where that's needed because the rootfs/overlay is re-created. Cases where sysupgrade does not destroy the (overlay) filesystem contents also exist and obviously config files aren't lost than.
> 
> Is there any case where rootfs/overlay is not destroyed? Even for x86,
> sysupgrade uses dd.
When overlay is mounted from an external disk ?

> 
>> After/during a sysupgrade a normal step is the restoration of the previous configuration, i.e. of config files *and* extra previously installed packages - currently sysupgrade does not handle re-install of extra user installed packages. This functionality could also serve cases where a certain configuration is to be put on a new or other device. Where the filesystem its contents is not lost during sysupgrade, this would just amount to an opkg upgrade of all outdated packages after sysupgrade has upgraded the kernel and possibly the rootfs.
>> 
>> In the more common sysupgrade case though (overlay contents are lost), some means is needed to preserve configuration data in order to be able to restore the configuration. Where this data is preserved during sysupgrade depends completely on the device specific sysupgrade mechanism; *what* data is saved and *how* it is re-applied is preferably opaque to, and independent of, sysupgrade.
> 
> IMHO, backup should be a different script. It's strange to ask
> sysupgrade for a backup (and worse to restore one). It might be just
> legacy calling for a refactoring.
I'm not sure I do understand correctly: what makes legacy a different use-case ?
But indeed, it is strange to implement back-up in sysupgrade.
It is not strange though when sysupgrade invokes some CM functionality to preserve the configuration over the sysupgrade; that has little to do with what is normally understood to be back-up: some functionality against data loss that stores a copy of some data on a secondary storage from which it can be restored.
Anyhow, the idea is that separate CM functionality exists that handles what and how, which sysupgrade can invoke with as argument where to store/load the configuration data to be preserved.

> 
>> Also, during upgrades new versions of packages may require updates to the config files. Preserving customisations of the config files cannot just be handled with back-up/restore and may require applying a diff or user intervention. How to deal with that can really only be determined by the package maintainer and so must be handled by the package itself. Anyhow, this clearly is in the domain of opkg and applies to upgrades outside of sysupgrade as well.
> 
> RPM, as opkg, simply saves the files and let user solve it manually.
> Doing it automagically will require too much resource from maintainers
> and the device that we might not have.
Clear. Of coarse it would still be convenient when, given that the changes to the distributed config file allows just automatically re-applying the customisations made by the user (patch of diff), that CM/opkg would have a hook for, and otherwise would let the user handle this as currently done.

> I guess we have what we can get. CM might replace the "package
> installation and configuration" part of the backup but it is not a
> backup. You still might need to save some data.
> For me, backup and CM are different beasts.
Totally agree on that.
The CM dataset (config files and user selection of extra packages) could be backed up though.

> 
>> In short, factoring out the configuration management functionality (CM) seems a good idea. The sysupgrade subsystem handles with upgrading kernel and rootfs and may invoke CM functions to preserve the user configuration. CM can could also take care of user installed packages. CM calls opkg to install and/or upgrade additional packages. Truly general back-up (e.g. as a defense against data loss) is outside the scope of sysupgrade.
>> 
>> Quite a few efforts have already been made toward some kind of CM, notably by offering this as an external service. The task to factor out generic CM  functionality that can be called upon in different contexts is probably quite complicated and comments on that idea are welcome.
>> 
>> What would be a practical API, if any, for CM ?
>> What (minimum) configuration data would be needed by CM, i.e. what's needed besides config files to store user install selections ?
>> What's needed to enhance opkg so that it can handle in a general way updates to config file while preserving user settings ?
>> Compatibility issues ?
>> 
>> Paul
> 
> I just opened a ticket for improving opkg
> (https://bugs.openwrt.org/index.php?do=details&task_id=1821).
Good, lets continue there.

> 
> Another great improvement would be to require that every single file
> in /etc/config should be uci generated. Today, some files are manually
> edited. So, when (l)uci touches them, every single line is changed,
> making diff simply useless. Would it be possible to enforce that rule
> in OpenWrt? OpenWrt-packages could do that with travis (uci import
> conf1/uci export conf1 to conf2/fail if conf1 and conf2 differ). It
> will, although, prohibit any comments in config files from packages.
Right you are. A general option named comment might fill the gap ?

> 
> My patch and these suggestions try to reduce the burden of an upgrade.
> Instead of implementing CM, it might be more useful to have tools that
> help with the config merging process, like a vimdiff-like app for Luci
> (remember, some users does not know how to use CLI).
A bottom-up approach is practical and may serve other purposes also. Whether CM is a named and well defined component or is achieved by combinations of some generic functions is not that relevant. Having also a function that yields a list of the extra packages selected by the user (upgrades and dependencies excluded) would be most helpful.

> 
> Regards,


_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel