[LEDE-DEV] overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Miklos Szeredi]

On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard at nod.at> wrote:
> Rafał,
>
> On 06.09.2016 12:46, Rafał Miłecki wrote:
>> Hi,
>>
>> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
>> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
>> overlayfs to combine these two.
>>
>> 1) Before:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>
>> 2) Mounting overlayfs
>> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>>
>> 3) After:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>>
>> It works fine as long as there aren't unexpected power cuts. I can easily break
>> it in a well-reproducible way by doing:
>> date > bar.txt && rm foo.sh
>> sleep 2s
>> [power cut]
>>
>> On the next boot LEDE/OpenWrt script calls the same mount function but it
>> results in:
>> [   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>>
>> I tracked that error and got a following forward trace for it:
>> ovl_fill_super
>> ovl_workdir_create
>> ovl_cleanup
>> vfs_rmdir
>> ubifs_rmdir
>> check_dir_empty
>>
>> In other words mounting overlayfs fails because ubifs refuses to remove
>> work/work as this directory is not empty. On a clean boot /overlay/work/work is
>> empty and everything works fine but after a power cut I can see
>> /overlay/work/work/#c752f2a8 file.
>>
>> I don't really know anything about overlayfs and its workdir, I just expected
>> overlayfs to recover from such situation on its own. FWIW after a power cut and
>> reboot my bar.txt is an empty file and foo.sh still exists.
>>
>> Does it sound like an overlayfs issue to you? Can you think of some way of
>> fixing this?
>
> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
> workdir yourself before mounting.
> Miklos?

Fixed in 4.8-rc5 by

eea2fb4851e9 ("ovl: proper cleanup of workdir")

I think it's better if kernel does the cleanup, but if it's easier to
upgrade userspace, then that's a valid workaround.

Thanks,
Miklos