[PATCH] fs: remove O_DIRECTORY flag from O_TMPFILE

[Stefan Kerkmann]

Decompressing fit images fails since release 2025.05.0 with commit
a1248198fa ("fs: implement O_DIRECTORY").

The reason is that converting the decompression buffer fd back to a file
in the `fd_to_file` fails the `O_DIRECTORY` flag check. This is flag is
set because `O_TMPFILE` is defined to contain the `O_DIRECTORY` flag.

The `O_TMPFILE` define is copied from the linux kernel and was
introduced in commit bb458c644a59d ("Safer ABI for O_TMPFILE"), the
reasoning behind including `O_DIRECTORY` is to prevent bugs in callers
by enforcing backwards incompatibility[1] with older kernels.

Barebox doesn't need that workaround as:

1. The barebox releases are atomic and all applications are compiled in
   thus backwards (in)compatibility isn't an issue.
2. With this commit openat function checks that the path passed is a
   directory e.g. a named file `/tmp/xyz` opened with `open("/tmp/xyz",
   O_TMPFILE | O_RDWR)` will fail with `ENODIR`.

[1]: https://lore.kernel.org/all/8738rk9eai.fsf@rasmusvillemoes.dk/

Signed-off-by: Stefan Kerkmann <s.kerkmann at pengutronix.de>
Fixes: a1248198fa ("fs: implement O_DIRECTORY")
---
 fs/fs.c         | 10 ++++++++--
 include/fcntl.h |  4 +---
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/fs/fs.c b/fs/fs.c
index 465fd617c2baa268ea3e5da784d31436629abd51..d401268c1e3420c32037e1a48d1f516d03cd61de 100644
--- a/fs/fs.c
+++ b/fs/fs.c
@@ -2552,8 +2552,14 @@ int openat(int dirfd, const char *pathname, int flags)
 	struct dentry *dentry = NULL;
 	struct path path;
 
-	if ((flags & O_TMPFILE) == O_TMPFILE) {
-		fsdev = get_fsdevice_by_path(dirfd, pathname);
+	if (flags & O_TMPFILE) {
+		error = filename_lookup(dirfd, getname(pathname), LOOKUP_DIRECTORY, &path);
+		if (error)
+			return errno_set(error);
+
+		fsdev = get_fsdevice_by_dentry(path.dentry);
+		path_put(&path);
+
 		if (!fsdev) {
 			errno = ENOENT;
 			return -errno;
diff --git a/include/fcntl.h b/include/fcntl.h
index f3694c2bba5f418ff018e0a694480775b2a89ece..db7926ee25fbe14607063d420697964801cd8321 100644
--- a/include/fcntl.h
+++ b/include/fcntl.h
@@ -33,12 +33,10 @@
 #define O_RWSIZE_4	004000000
 #define O_RWSIZE_8	010000000
 
-#define __O_TMPFILE	020000000
+#define O_TMPFILE	020000000	/* open as temporary file in ramfs */
 #define O_PATH		040000000	/* open as path */
 #define O_CHROOT	0100000000	/* dirfd: stay within filesystem root */
 
-#define O_TMPFILE       (__O_TMPFILE | O_DIRECTORY)
-
 #if IN_PROPER
 int openat(int dirfd, const char *pathname, int flags);
 #else

---
base-commit: 8e219a9cda0f5d6145d267e27a01ff0cd828e4c8
change-id: 20250604-fix-tmp-file-90367b6febfe

Best regards,
-- 
Stefan Kerkmann <s.kerkmann at pengutronix.de>