[PATCH RFC 1/3] fs: add open O_TMPFILE support

[Ahmad Fatoum]

barebox dentry cache is never cleared with the assumption that there
should be enough RAM anyway to cache all lookups until boot.

When fuzzing barebox however, there is no limit to how many dentries
are added to the cache. This is e.g. problematic when fuzzing the FIT
parser: FIT images can have compressed payloads. Compressed payloads are
passed to uncompress_buf_to_buf, which uses a new random file in ramfs
as destination. A fuzzer would thus create a dentry for every iteration,
rapidly depleting memory.

A general solution for that would be dropping the dentry cache on memory
pressure. In the special case of uncompress_buf_to_buf, it would already
be enough though to sidestep the dentry cache and create an anonymous
file. Linux provides this with the O_TMPFILE option, so let's add the
equivalent to barebox.

Signed-off-by: Ahmad Fatoum <a.fatoum at pengutronix.de>
---
 fs/fs.c         | 29 +++++++++++++++++++++++++++++
 include/fcntl.h |  3 +++
 2 files changed, 32 insertions(+)

diff --git a/fs/fs.c b/fs/fs.c
index 1800d6826ddc..6bd3c2df3c31 100644
--- a/fs/fs.c
+++ b/fs/fs.c
@@ -2539,6 +2539,35 @@ int open(const char *pathname, int flags, ...)
 	const char *s;
 	struct filename *filename;
 
+	if (flags & O_TMPFILE) {
+		fsdev = get_fsdevice_by_path(pathname);
+		if (!fsdev) {
+			errno = ENOENT;
+			return -errno;
+		}
+
+		if (fsdrv != ramfs_driver) {
+			errno = EOPNOTSUPP;
+			return -errno;
+		}
+
+		f = get_file();
+		if (!f) {
+			errno = EMFILE;
+			return -errno;
+		}
+
+		f->path = NULL;
+		f->dentry = NULL;
+		f->f_inode = new_inode(&fsdev->sb);
+		f->f_inode->i_mode = S_IFREG;
+		f->flags = flags;
+		f->size = 0;
+		f->fsdev = fsdev;
+
+		return f->no;
+	}
+
 	filename = getname(pathname);
 	if (IS_ERR(filename))
 		return PTR_ERR(filename);
diff --git a/include/fcntl.h b/include/fcntl.h
index 2e7c0eed3479..1b4cd8ad3783 100644
--- a/include/fcntl.h
+++ b/include/fcntl.h
@@ -16,6 +16,9 @@
 #define O_APPEND	00002000
 #define O_DIRECTORY	00200000	/* must be a directory */
 #define O_NOFOLLOW	00400000	/* don't follow links */
+#define __O_TMPFILE	020000000
+
+#define O_TMPFILE       (__O_TMPFILE | O_DIRECTORY)
 
 /* barebox additional flags */
 #define O_RWSIZE_MASK	017000000
-- 
2.39.2