Conclusions from CVE-2024-3094 (libxz disaster)

[Daniel Golle]

Hi everyone!

you may all have heard and read about CVE-2024-3094. If not, please do
so now [1], [2].

This incident has exposed many long standing issues and should not be
seen as a singular event, but rather as the result of several
unhealthy patterns. And while OpenWrt was not affected by the
resulting vulnerability (because we don't ship OpenSSH by default; and
our build of OpenSSH isn't downstream patched for integration with
systemd...), this and the fact that the added backdoor was discovered
rather fast are both just lucky coincidents.

All of that comes in a moment that the NVD [3] has practically stopped
working with the public [4], at least temporarily, and that has
severely increased the pressure on open source projects to handle such
incidents responsibly **by themselves**. (how?)

Of course there are many conclusions to draw and discussions to be had
when it comes to the libxz bootdoor incidence, many on the purely
technical level (why did Debian and RedHat downstream-patch OpenSSH?
Why did a dependency-monster like systemd become the de-facto standard
on Linux installations? ...), and all of them need to be re-evaluated
in the light of this attack imho. In many ways, we are already better
off than most Linux distros out there -- not because of deliberate
decisions with security in mind, but because of our tendency to
minimalism and avoiding bloat due to resource limitations on the
target devices, and having a reduced attack surface is just an
indirect consequence of that.

However, after reading up about the details of this backdoored release
tarball, I believe that the current tendency to use tarballs rather
than (reproducible!) git checkouts is also problematic to begin with.

Stuff like 'make dist' seems like a weird relic nowadays, creates more
problems than it could potentially solve, bandwidth is ubiquitous, and
we already got our own tarball mirror of git checkouts done by the
buildbots (see PKG_MIRROR_HASH). So why not **always** use that
instead of potentially shady and hard to verify tarballs?

Why do we need to rely one proprietary hacks such as Gibhub codeload
just to safe a few megabytes of traffic and a few seconds of build
time?

There are even too many problems to reproduce even those supposedly
automated Github-generated tarballs. Nobody actually checks that.
9bd7d8b, c7c2257, 77368ec, 86994e1, 954142f, 4c5d910, 21f713d, ...
Probably all of those have trivial causes and there isn't anything
malicious going on there. But it shows the general problem: That while
we are quite good with (much more complicated) reproducible builds,
nobody seems to care about (actually trivial) reproducible **sources**.
It's too trivial and boring to care, I suppose.

And then there is the not exactly glorious role of Github in that whole
mess, blocking the affected repo (making analysis harder or impossible),
blocking the account of the semi-retired old maintainer who is not to
blame, ...

Hence I believe we should use git checkouts and reference to (ideally
signed) git tags when ever we can. Package maintainers should always
have a local checkout of the repository of the software they are
packaging for OpenWrt, and use 'git fetch origin' or 'git pull' to
keep it up-to-date, as to make sure that the repo history remains
unchanged. Git has a lot of security built-in, and by using tarballs
as a base for our package builds we are basically throwing all that
away, for the sake of saving a negligible amount of resources on
the build infrastructure.

Always using git checkouts instead of tarballs would also makes it
much easier for maintainers to at least have a quick look at the
changes made in an upstream project between versions (a quick scroll
over  'git diff oldtag..newtag' or even just 'git log --stat
oldtag..newtag' doesn't take much more time than manually validating a
release tarball GPG signature in most cases, if there even is any...).

Hiding a malicious change in a commit is infinitely harder than hiding
it in a tarball.

Those are just my 2 cents, I think this cries for a wider debate and
I encourage everyone to participate in it.

Of course, the real problem (which is much deeper) is the lack of
maintainer resources for critical infrastructure. This message [5]
says it all.


Cheers


Daniel

[1]: https://www.openwall.com/lists/oss-security/2024/03/29/4
[2]: https://boehs.org/node/everything-i-know-about-the-xz-backdoor
[3]: https://nvd.nist.gov/
[4]: https://www.helpnetsecurity.com/2024/03/19/nvd-vulnerability-management/
[5]: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html