[LEDE-DEV] [PATCH] base-files: do not backup unchanged files

[Matthias Schiffer]

On 11/24/2017 08:11 AM, Luiz Angelo Daros de Luca wrote:
>>> After applied, the list of files in backup in a fresh installation change from:
>>> to:
>>>
>>> root at LEDE:~# sysupgrade  -l
>>> /etc/config/dhcp
>>> /etc/config/firewall
>>> /etc/config/network
>>> /etc/config/system
>>> /etc/dropbear/dropbear_rsa_host_key
>>>
>>
>> Hi,
>>
>> when I set up  new devices I usually do an export of the initial configuration to track
>> the changes I made externally in a git repo.
>> With this patch I would just get the five files mentioned above, thus an incomplete config and would not
>> be able to track my config changes.
> 
> Hi,
> 
> Yes, a diff between two backups will not show actually what has changed as modified files will appear as new.
> 
> However, maybe a configuration backup is not the best way to keep track of changes on configuration files.
> The list of "configuration file" can change as long as their configuration are not listed in /lib/upgrade/keep.d.
> Backup consists of files listed in /etc/sysupgrade.conf, /lib/upgrade/keep.d and changed config files (tracked by opkg).
> For example, /etc/firewall.user will not appear in backup if not modified (or opkg is missing) because it is not mentioned in
> /lib/upgrade/keep.d. There are many more examples. Try yourself:
> 
> # find $(cat /lib/upgrade/keep.d/*) -type f -o type l 2>/dev/null | sort -u >/tmp/a.txt
> # cat /usr/lib/opkg/info/*.conffiles | sort -u>/tmp/b.txt
> # diff -u /tmp/a.txt /tmp/b.txt  | grep +
> 
> So, even today, it is already expected that a backup after fresh installation to have less files than a backup
> after some packages where installed. I just made the difference bigger.
> 
> BTW, the backup is broken whenever opkg is missing, as those files will never get into backup. I don't know if it's worth it to try
> to fix backup without opkg.

I'll eventually get to it, as the LEDE-based Gluon mesh framework generally
ships without opkg, and without most of /usr/lib/opkg, on small (4MB)
devices. I plan to ship a minimal /usr/lib/opkg that just contains the
config hashes, and include a small replacement for `opkg
list-changed-conffiles` in sysupgrade).


> 
> I could add a new option for sysupgrade in order to keep all files listed, even those untouched, in order to get the old behaviour.
> However, in order to use it, you'll need to call it using a terminal. In this case, you could also retrieve those files using ssh+tar, or rsync if avaiable.
> Anyway, this option would only make sense if I grab also all config files mentioned in any installed package, even those untouched.
> 
> I'm trying to make OpenWrt/LEDE upgrade process closer to what we get in a normal Linux distribution. The inclusion of only
> modified files is a trick to reproduce the same logic rpm/deb uses for configuration updates (replace what wasn't touched,
> keep what was modified). The current behaviour is already breaking installations by, at least, propagating outdated /etc/profile.
> I'm trying to fix backup/upgrade process from now on, even if it could "break" the use of backup for things not related to restore/upgrade.
> IMHO, it pays off.

As mentioned before, I don't like the idea of checking /rom to find out
what has changed. It will make sysupgrade behave differently on devices
that don't have an overlay and thus no /rom.

A better fix (that would also be closer to what "normal Linux
distributions" do) would be to reduce the use of /lib/upgrade/keep.d, and
instead rely on opkg's config hash tracking for a lot more files (we should
make this work without opkg itself first though). This would also work with
Moritz's usecase of initial config files supplied in the image build via
/files, as these files would not match the hashes known to opkg.

/lib/upgrade/keep.d should still be supported IMO, it is useful for adding
directories and files to the backups that generated dynamically (Gluon has
a few usecases for this, e.g. packages that clean up behind themselves by
adding cleanup scripts that remove old configuration on the first boot with
a new image that doesn't have the package using the configuration anymore).
As /lib/upgrade/keep.d would mostly be used for files that don't exist in
/rom anyways, we can keep the behaviour of backupping these files
unconditionally, and don't need to introduce (more) overlay-specific hacks.


> 
> I already have a patch v2 ready https://github.com/luizluca/source/tree/better-backups_skip-unchanged.
> I'm just waiting because it is now based on the submitted "[PATCH] base-files: fix sysupgrade -b/-l when -c is used".
> Or maybe I should just send already v2 including both.
> 
> Luiz


Thanks for bringing this up. I generally agree that our backup strategy
could use some mending, but I'd prefer to keep /rom hacks out of our
scripts as much as possible.

Matthias

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/lede-dev/attachments/20171124/ecf37724/attachment.sig>