[PATCH v1 0/5] arm64/mm: uffd write-protect and soft-dirty tracking

[Ryan Roberts]

On 23/04/2024 22:02, David Hildenbrand wrote:
>>>
>>> Shivansh, do you speak for CRIU? Are you able to comment on whether CRIU
>>> supports checkpointing an app that uses uffd?
>>
>> I do not speak for CRIU - I'm just a user (and hopefully a future
>> contributor), but not a maintainer or owner. I can however comment on whether
>> CRIU supports checkpointing an app that uses UFFD - it doesn't. Looking
>> through both the implementation of CRIU (specifically how they restore memory
>> [1]), and at recently filed Github issues [2], it's pretty clear that CRIU
>> doesn't support processes using UFFD - that they do not currently have plans
>> to [3].
> 
> Thanks for all these pointers!
> 
>>
>> [1]
>> https://github.com/checkpoint-restore/criu/blob/criu-2.x-stable/criu/mem.c#L683
>> [2] https://github.com/checkpoint-restore/criu/issues/2021
>> [3]
>> https://github.com/checkpoint-restore/criu/issues/2021#issuecomment-1346971967
>>
>>>>
>>>> Further ... isn't CRIU already using uffd in some cases? ...documentation
>>>> mentions [1] that it is used for "lazy (or post-copy) restore in CRIU". At
>>>> least
>>>> if the documentation is correct and its actually implemented.
>>>>
>>>
>>> Shivansh, same question - do you know the current CRIU status/plans for using
>>> uffd-wp instead of soft-dirty? If CRIU doesn't currently implement it and has no
>>> current plans to, how can we guage interest in making a plan?
>>>
>>
>> While I cannot gauge whether the maintainers or main contributors of CRIU plan
>> on using uffd-wp instead of soft-dirty in the future, I can tell you that
>> there is no currently open issue to track that work, and whenever anyone in
>> the past has asked about ARM64 pre-dump support to CRIU (which is the feature
>> that uses soft-dirty/would use uffd-wp), they've always just said it's not
>> supported - but that they do want the feature [4].
>>
>> So in summary, they want the feature, but no one is working on implementing it
>> (either with soft-dirty or with uffd-wp).
>>
>> I doubt that CRIU would have any issues with adding the feature via soft-dirty
>> (since, as shown in [4] they're interested in it), but as for using uffd-wp
>> they definitely haven't shown any interest thus far. Based on the fact that it
>> would be a very significant amount of work and it would really only be for
>> ARM64 support (which they're already fine without), I'd be very surprised if
>> they were interested in pursuing it.
>>
> 
> Of course, nobody wants to do the work. But that doesn't mean that the kernel
> has to do the work :)
> 
> If there are some major challenges why it cannot possible be done with uffd-wp
> (unfixable), that's a different story.
> 
>> [4]
>> https://github.com/checkpoint-restore/criu/issues/1859#issuecomment-1972674047
>>
>>>>
>>>>>
>>>>>> But I'll throw in another idea: do we really need soft-dirty and uffd-wp to
>>>>>> exist at the same time in the same process (or the VMA?). In theory, we
>>>
>>> My instinct is that MUXing a PTE bit like this will lead to some subtle problems
>>> that won't appear on arches that support either one or both of the features
>>> independently and unconditionally. Surely better to limit ourselves to either
>>> "arm64 will only support uffd-wp" or "arm64 will support both uffd-wp and
>>> soft-dirty". That way, we could move ahead with reviewing/merging the uffd-wp
>>> support asynchronously to deciding whether we want to support soft-dirty.
>>>
>>
>> My personal preference is having both approaches supported - especially in the
>> context of CRIU since I doubt they'll be willing to rewrite all of the dumping
>> and restore logic just for ARM64 support.
> 
> Sure, nobody does any work unless they are forced to.
> 
> But this is something that arm64 maintainers will have to decide.
> 
> Let's start with uffd-wp that has other well-known users that could benefit
> (e.g., QEMU background snapshots).

Right. I'm going to:

  - re-post patch 5 standalone to go in via kselftests.
  - re-post patches 1 & 2 as a series to enable uffd-wp on arm64; uncontentious
    I think.
  - Have a chat with Catalin about appetite for soft-dirty on arm64; But likely
    that will be left here until/unless there is clear justificaiton that the
    use case cannot be met with uffd-wp.

Thanks,
Ryan