[LEDE-DEV] [RFC] filesystem-agnostic sysupgrade backup system

[Thibaut VARÈNE]

Hi,

Following up on [1], I propose a different way of handling sysupgrade backups that would no longer rely on a filesystem stub (jffs2, ubifs or otherwise).

Note: the following elements describe squashfs with jffs2 overlay images, but I suspect they apply to more cases (ubifs…).

Current status:
- When sysupgrade is asked to preserve the device configuration over reflash, the process will:
	- prepare a tarball of the config data onto tmpfs
	- embed it into a filesystem stub appended after the squashfs data
	- append a specific marker (deadc0de) to designate the end of the valid jffs2 segment
- When the new image boots:
	- the squashfs mtdsplitter locates the end of the squashfs image and places the beginning of the rootfs_data partition after it
	- the jffs2 stub is now located at the beginning of the rootfs_data partition (the writeable overlay)
	- a LEDE-specific patch to the jffs2 driver clears the nodes located after the marker
	- the overlay is then mounted and the data it contains is recovered.

This has several drawbacks, as [1] illustrates: in particular it makes it complicated, if not impossible, to change the filesystem used on the overlay or any of its parameters - such as sector size in the case of jffs2 - without losing the backup data.

Proposal:
Use a plain archive on the mtd.

Implementation detail:
- When sysupgrade is called to preserve the device configuration over reflash, the process will:
	- prepare a well formatted (see below) archive of the configuration data onto tmpfs
	- append that data after the squashfs image on the mtd
- When the new image boots:
	- the squashfs splitter locates the end of the squashfs image and places the beginning of the rootfs_data partition after it (step unchanged)
	- the archive is now located at the beginning of the rootfs_data partition
	- a hook to the preinit process will attempt to unpack that archive into tmpfs
		- if it is successful (archive found and valid), the rootfs_data partition will be formatted
		- if it is unsuccessful (archive found but invalid), the rootfs_data partition will be formatted
		- if it is unsuccessful (archive not found), the rootfs_data will not be formatted (regular boot)
	- the overlay will then be mounted and if present the data from the archive will be applied (see below)

Archive format:
For the sake of simplicity and futureproofness, I propose:
- that the archive data be plain, uncompressed tar (tar has low overhead, has checksumming and was designed to operate in the absence of a filesystem)
- that the archive content be prepared as follows:
	- sysupgrade.tgz (the current compressed backup tarball that is stored in the jffs2 stub)
	- a ‘payload’ file, which would be a script containing two mandatory elements:
		- a “REVISION” variable to store the current version/revision of the system that generated the backup
		- a “restore()” function that would be called to restore the content of the backup
	The ‘payload’ file would be the last element in the tar archive

Goals:
- Preserve the ability to use various compression schemes in the archive data without imposing ex-ante knowledge of it from the new firmware image (decompression is handled by the ‘payload’ script itself).
- Ensure that if the archive is truncated or damaged, the ‘payload' script will not be created and thus not executed (the payload script could perform final checksumming of the data and of itself too, if necessary).
- Preserve future flexibility in the content of the archive data, enabling other (platform-specific?) hooks to be added down the road without breaking backward compatibility (by building into the ‘payload’ script) and not enforcing any specific data structure other than the presence of the ‘payload’ script.
- Enable the new firwmare to easily detect what version the archive data comes from and make use of that information if necessary (e.g. apply special treatment or discard the backup).

Benefits:
- No longer tie backup data to a specific filesystem, enabling seamless change to the underlying overlay filesystem.
- Full control over rootfs_data segment: ensure overlay integrity after flash by performing a true format.
- Get rid of all out-of-tree, LEDE-specific hacks to jffs2 etc, making it possible to use vanilla code instead.
- Flexible, futureproof system.

Notes:
- The proposed system does put more strain on the RAM usage during upgrade than the existing system.
- The proposed system mtd space requirements are similar to the existing system (same compression level, one extra file, overhead of tar can be lower than jffs2)
- In the case of factory images, the deadc0de marker could simply be replaced by an archive containing a single ‘payload’ script whose sole purpose would be to trigger the overlay formatting.

The proposed system should be possible to implement alongside the existing one if a transition is desirable (if no archive is found in rootfs_data, fall back to the current system of looking for the deadc0de marker).

Would there be interest in such a system? Have I missed something?
Comments welcome :)

T.

[1] http://lists.infradead.org/pipermail/lede-dev/2017-November/009863.html