[LEDE-DEV] [PATCH] base-files: do not backup unchanged files

[Luiz Angelo Daros de Luca]

Kevin,

> You need to bump PKG_RELEASE in base-files/Makefile

I missed that one. I'll fix it.


Stefen,

> Is this really needed? Just try to remove package/base-files/files/lib/
upgrade/keep.d/base-files-essential and your problem will be gone. I think this 
file should be only added to the image if opkg is not installed.
>
> https://lists.openwrt.org/pipermail/openwrt-devel/2013-February/018742.html

Removing base-files-essential will only "solve" the problem for those files 
listed there (but not for /etc/sysctl.d/local.conf). I'm trying to change the 
logic and not the input of "what is interesting for backup".


Koen,

> Did you also account that some of the files will still be backupped (even 
when unchanged) as they are also referenced here? :
>
> [ Casino BB | node-10 ] cd /lib/upgrade/keep.d/

AFAIK, this patch applies to all files mentioned /etc/sysupgrade.conf, /lib/
upgrade/keep.d/ or opkg list-changed-conffiles


Matthias,

> I don't like adding overlay-specific logic here, sysupgrade should work
> "the same" on all targets, with or without overlay. Maybe we can rather
> filter out unchanged files *after* the sysupgrade (after a major config
> backup rework I guess - but I heard people are thinking about a generic
> tar-based backup approach for sysupgrades).

 it really depends on /rom, which is present when overlay is in use.
However, if space is not a problem, a simple copy of pristine / into /rom 
would work.
In the future, OpenWrt might use a FS that has snapshots, which could also 
provide
/rom.

> Another issue with this patch: it breaks for filenames with spaces in the
> opkg list-changed-conffiles output (might or might not be an actual issue,
> but something to keep in mind).

Yes, I'll break. I noticed that but the only solution that I imaged consists 
of a
hacky multiline script like: "set --; while read line; do set -- '$@" "$line"; 
done <"$output of opkg..."
I'll fix it.


Jonas,

> 1) The hashing of opkg seems to be broken:

Sorry, I don't have this problem. Anyway, the patch helped fix a bug (bonus).

> 2.) We *do* modify some files during offline root generation after the
base package had been installed.

I don't think this as an issue. Build process do change files but I imagine 
that a sysadmin
might also want to generate a custom image with changed files. IMHO, packages 
must
still consider those files changed.

> The question for implications is if there are parts that rely on files
> being implicitly backed up, and can there be issues when only doing
> partial backups. On a first glance it seems that adding a user to an
> existing group will still modify the /etc/groups, so /etc/passwd and
> /etc/users will always be backed up at the same time, thus group ids
> cannot go out of sync (in case they are dynamically allocated).
>
>> - but the files
>> listed in /etc/sysupgrade.conf or /lib/upgrade/keep.d are backed up
>> unconditionally. I'm not sure which behaviour the average admin would
>> expect - but I'm not happy about the different handling of config files
>> managed by opkg and the other file lists.
>>
>> I don't like adding overlay-specific logic here, sysupgrade should work
>> "the same" on all targets, with or without overlay. Maybe we can rather
>> filter out unchanged files *after* the sysupgrade (after a major config
>> backup rework I guess - but I heard people are thinking about a generic
>> tar-based backup approach for sysupgrades).
>
> This is IMHO a bit of a philosophical question - if one specifies
> additional files to backup, does this mean we should back them up in
> any case, or should we second guess the user and check if they were
> present before and were modified since then? Maybe we need a new flag
> for that (meaning "full config backup" or "modified-only").

I'm having difficult to think on a case where the sysadmin would like to copy a 
file
that was already in the original image into the new system.

I do use a different modified-only backup solution that depends on overlay and 
opkg.
I'm cleaning it up before sending the patch. It simply backups all files in 
overlay,
except for those files that comes from packages, but including "opkg list-
changed-conffiles" ones.
It is a "'sysupgrade -c' on steroids". This overlay backup together with a 
batch installation of all
extra packages previously installed was able to restore my most complex 
installation with no extra intervention.

> I have to admit, I always assumed that if I tell sysupgrade to backup
> the config, it backs up the full config, and not only the modified
> parts. But that might be only me ;-)

If the user asked to backup a file, I agree that it should be in the backup. 
Maybe
/etc/sysupgrade.conf should be included unconditionally. However, the user
never asked to include /etc/profile. It is the default behavior from keep.d 
files.

I got into this because a custom script that depends on /etc/profile.d simply 
failed for some
LEDE routers. Same HW, same LEDE version, different behavior. I only isolated 
the problem
after doing a full FS diff, detecting that /etc/profile was still from OpenWRT 
BB era.
This is not how a "normal" linux distribution would work. RPM/DEB will quietly 
replace unchanged
config files on an upgrade. In the case of /etc/profile, as it is more code than 
config, I guess they
will quietly replace even if modified.

BTW, this is an open issue (but not reported until now). Any software that 
depends on /etc/profile.d
will not work on any LEDE system upgraded from OpenWRT because /etc/profile is 
outdated. An old
/etc/profile gave me headaches but other outdated files in backup list might 
break other things for other sysadmin.

This patch will not fix that but it could prevent new cases.

> Although, the help says:
>
>         -b | --create-backup <file>
>                      create .tar.gz of files specified in sysupgrade.conf
>                      then exit. Does not flash an image. If file is '-',
>                      i.e. stdout, verbosity is set to 0 (i.e. quiet).
>
> and the sysupgrade.conf says:
>
>   ## This file contains files and directories that should
>   ## be preserved during an upgrade.
>
> Which sort of implies to me that it creates a full backup - it doesn't
> mention anywhere that it will skip unmodified files. Not that I'm
> against doing the diff-version, just that the texts should be updated
> as well in that case.

Sure, they need to be updated as well.

Regards,

Luiz Angelo