make modules_prepare failing

[Tim Harvey]

On Wed, Jun 16, 2021 at 12:55 PM Russell King (Oracle)
<linux at armlinux.org.uk> wrote:
>
> On Wed, Jun 16, 2021 at 12:35:16PM -0700, Tim Harvey wrote:
> > Greetings,
> >
> > I'm trying to understand how to use the 'modules_prepare' target to
> > prepare kernel headers on a dev host appropriate for building
> > out-of-tree modules on a target board. I'm interested in the minimum
> > headers necessary and have noticed the Ubuntu 'linux-headers-*'
> > packages are only about 24MiB.
> >
> > I've looked over
> > https://www.kernel.org/doc/Documentation/kbuild/modules.txt and it
> > does not provide an example showing how to build the headers in a
> > different directory.
> >
> > $ make O=foo modules_prepare
> > make[1]: Entering directory '/usr/src/venice/bsp/linux/foo'
> > ***
> > *** The source tree is not clean, please run 'make mrproper'
> > *** in /usr/src/venice/bsp/linux
> > ***
> > /usr/src/venice/bsp/linux/Makefile:512: recipe for target
> > 'outputmakefile' failed
> > make[1]: *** [outputmakefile] Error 1
> > make[1]: Leaving directory '/usr/src/venice/bsp/linux/foo'
> > Makefile:179: recipe for target 'sub-make' failed
> > make: *** [sub-make] Error 2
>
> This means your tree (/usr/src/venice/bsp/linux) is not clean, it
> is not referring to "foo". It is only possible to do a split source/
> object tree build with a clean source tree.
>
> So, this is telling you to do "make mrproper" with nothing else in
> the source directory - no O= argument. If that is still failing,
> then it suggests there is a file somewhere in the source tree that
> shouldn't be there.
>
> > The 'modules_prepare' does work if I don't try to output to a
> > different directory but then it isn't clear what to package compared
> > to the entire >1GiB kernel source tree.
>
> I'm not sure you're going about this in the correct way. Doing a
> split object tree build does not mean everything you need to build
> modules is in the object tree - much depends on the source tree for
> the kernel build infrastructure and the other header files in the
> include subdirectory, as well as other places (such as picking up
> architecture options from arch/*/Makefile and the architecture
> specific header files.)
>
> So, if you are thinking "I can just package up the object tree to
> build modules" that is not correct.
>
> > I wish https://www.kernel.org/doc/Documentation/kbuild/modules.txt had
> > an example expalining how a distro packager would create
> > linux-headers-<foo>. I'm not familiar enough with Ubuntu packaging to
> > understand how to disect how their packages are created.
>
> Isn't that package used for building the C library rather than for
> modules? That will likely be created by "make headers_install" which
> gives you all the headers for userspace to use with kernel internals
> sanitised away.

Russell,

Thanks for the response.

I thought that Ubuntu's linux-headers-* package also provided enough
to build out of tree modules but perhaps I'm wrong. Here is what I did
to try and figure out what was in the linux-headers file on an Ubuntu
bionic system:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
$ uname -r
5.4.0-73-generic
$ dpkg -S /usr/src/linux-headers-5.4.0-73-generic
linux-headers-5.4.0-73-generic: /usr/src/linux-headers-5.4.0-73-generic
$ dpkg -L linux-headers-5.4.0-73-generic
/.
/lib
/lib/modules
/lib/modules/5.4.0-73-generic
/usr
/usr/share
/usr/share/doc
/usr/share/doc/linux-headers-5.4.0-73-generic
/usr/share/doc/linux-headers-5.4.0-73-generic/changelog.Debian.gz
/usr/share/doc/linux-headers-5.4.0-73-generic/copyright
/usr/src
/usr/src/linux-headers-5.4.0-73-generic
/usr/src/linux-headers-5.4.0-73-generic/.config
/usr/src/linux-headers-5.4.0-73-generic/.config.old
/usr/src/linux-headers-5.4.0-73-generic/.gitignore
/usr/src/linux-headers-5.4.0-73-generic/.missing-syscalls.d
/usr/src/linux-headers-5.4.0-73-generic/Module.symvers
/usr/src/linux-headers-5.4.0-73-generic/arch
...
$ dpkg -L linux-headers-5.4.0-73-generic | grep -v
"^/usr/src/linux-headers-5.4.0-73-generic"
/.
/lib
/lib/modules
/lib/modules/5.4.0-73-generic
/usr
/usr/share
/usr/share/doc
/usr/share/doc/linux-headers-5.4.0-73-generic
/usr/share/doc/linux-headers-5.4.0-73-generic/changelog.Debian.gz
/usr/share/doc/linux-headers-5.4.0-73-generic/copyright
/usr/src
/lib/modules/5.4.0-73-generic/build
$ ls -l /lib/modules/5.4.0-73-generic/build
lrwxrwxrwx 1 root root 39 Apr 16 07:44
/lib/modules/5.4.0-73-generic/build ->
/usr/src/linux-headers-5.4.0-73-generic
$ du -h /usr/src/linux-headers-5.4.0-73-generic/ | tail -n1
24M     /usr/src/linux-headers-5.4.0-73-generic/

When I started poking around to see how I could create this I did find
this guide here but it did not work (the whole mrproper clean issue):
https://unix.stackexchange.com/questions/270123/how-to-create-usr-src-linux-headers-version-files

What I am after is trying to see if I can easily appease users on
embedded Linux boards that want to build out-of-tree modules on their
target because they can't seem to wrap their head around building the
kernel+modules on a development host with a cross-toolchain (the
'proper' way to do it but which has a huge barrier to entry for many
users). They run into instructions online such as:

# build out-of-tree module
apt install build-essential
apt install linux-headers-$(uname -r)
# some out-of-tree modules have you point to kernel source via a non-std env var
make KSRC=/lib/modules/$(uname -r)/build
# others follow what the kernel docs have you do which is to do a make
in the kernel dir
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
make install

Best regards,

Tim