[LEDE-DEV] overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Rafał Miłecki]

Hi,

In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
rootfs mounted at / we mount rootfs_data volume at /overlay and then use
overlayfs to combine these two.

1) Before:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)

2) Mounting overlayfs
mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")

3) After:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)
overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)

It works fine as long as there aren't unexpected power cuts. I can easily break
it in a well-reproducible way by doing:
date > bar.txt && rm foo.sh
sleep 2s
[power cut]

On the next boot LEDE/OpenWrt script calls the same mount function but it
results in:
[   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only

I tracked that error and got a following forward trace for it:
ovl_fill_super
ovl_workdir_create
ovl_cleanup
vfs_rmdir
ubifs_rmdir
check_dir_empty

In other words mounting overlayfs fails because ubifs refuses to remove
work/work as this directory is not empty. On a clean boot /overlay/work/work is
empty and everything works fine but after a power cut I can see
/overlay/work/work/#c752f2a8 file.

I don't really know anything about overlayfs and its workdir, I just expected
overlayfs to recover from such situation on its own. FWIW after a power cut and
reboot my bar.txt is an empty file and foo.sh still exists.

Does it sound like an overlayfs issue to you? Can you think of some way of
fixing this?

-- 
Rafał