[RFC PATCH 00/14] driver core: defer per-VF sysfs creation for fast SR-IOV bring-up

[Sakac, Pavol]

On 7/14/26 01:50, David Matlack wrote:
Thank you for the feedback, I will try to answer in this thread.

> On Fri, Jul 10, 2026 at 7:16 AM Greg Kroah-Hartman
> <gregkh at linuxfoundation.org> wrote:
>>
>> On Thu, Jul 02, 2026 at 07:40:19PM +0200, Pavol Sakac wrote:
>>> Virtualization hardware keeps increasing in CPU-core and VF density.
>>> Kernel is trending towards preserving VFs using LUO across a kexec
>>> which will put SR-IOV in live-update hotpath. VFs must be re-added to
>>> the sysfs tree, along with its supporting devices (e.g. vfio,
>>> iommu_groups). On production hardware a single VF can consume ~80 kernfs
>>> nodes. With thousands of VFs on modern hardware, hundreds of thousands
>>> of kernfs nodes need to be created, which can add 100ms+ to the boot
>>> time and to guest downtime.
>>
>> Hundreds of thousands?  You have a hundred sysfs files per device?

It's more close to 80/VF - this includes all nodes that are in the end
created - device dir + vfio/iommu_group dirs for pass-through.

>>
>>> Proposed here is a PoC of deferred materialization of sysfs files until
>>> first access to avoid cost of kernfs node creation in hot path. In the
>>> reproducer below using a synthetic VF to isolate the sysfs overhead,
>>> the per-device sysfs-creation time is reduced by ~74%. It exploits the
>>> expected access pattern of hypervisors, where only a subset of device
>>> files need to be accessed to attach a VF to a VM on the hot path.
>>
>> Ick, that access pattern could change with other users, this feels very
>> fragile.

The assumption is that you have tight control of userspace and you'd 
prevent unfavorable access patterns.

>>
>>> To allow lazy init scheme, a minimal set of files and directories is
>>> built per device: a device directory resolves directly to the device's
>>> kobject and is used to materialize attributes at access time. The
>>> access is trapped in kernfs at the dir-read or path-walk stage and
>>> depending on the trigger, either the full directory or a single file is
>>> materialized.
>>>
>>> The changes are two logically distinct parts:
>>>
>>>   1. Refactor of hardcoded device_add / iommu_group sysfs attribute
>>>      creation to a table-driven form to allow walkability of all
>>>      attributes and single point of definition for attributes
>>>      regardless of lazy opt-in to avoid duplicate definitions and
>>>      associated maintainability issues with it.
>>
>> This is probably a good idea anyway.  Want to just send this as a series
>> to start with, so we can take that?
>>

Yup, sounds good. 

>>>   2. Lazy-init infra to front-run access to device files at lookup/
>>>      readdir time to trigger materialization. Opt-in per device: PCI VFs
>>>      and VFIO devices opt in via device_set_sysfs_lazy(); iommu_group
>>>      opts in at the kobject level.
>>
>> This is going to get complex.  Where exactly is the bottleneck?  I'm
>> guessing that systems with that many devices also have many hundreds of
>> CPUs, right?  Are we hitting a single lock somewhere?  There have been
>> changes in kernfs in the past to split up the locks to be more
>> fine-grained, perhaps that needs to continue to solve this?
> 
> +1 I also am wondering if the time can be eliminated entirely instead
> of deferred.
> 
> It would be nice to have a section of the cover letter that explains
> where the time is being spent and why lazy initialization was chosen
> over alternative approaches.
> 

The bottleneck is as I understand kernfs as it has a per root lock taken
for mutating operations. If granularity was at a directory level, we'd
have an option to perform adding VFs in parallel, though we might hit some
PCI config-space access contention. 

The lazy was selected as complexity of optimizing kernfs/parallel adding
seemed higher and benefit is that with control over access pattern, we can
also avoid the work entirely for nodes we don't ever access without
actually surgically removing them and maintaining the list.

>>
>>> As a PoC, the code does not yet follow this logical structure in the
>>> commit sequence and likely does not yet cover all corner cases.
>>>
>>> Synthetic reproducer measurements
>>> ---------------------------------
>>>   devices   per-dev time   total time       kernfs nodes (base -> lazy)
>>>   -------   ------------   --------------   ----------------------------
>>>      500    31.8->8.3 us    15.9-> 4.1 ms     36,417 ->  3,922  (72.8->7.8/dev)
>>>     1000    33.2->8.7 us    33.2-> 8.7 ms     73,427 ->  8,432  (73.4->8.4/dev)
>>>     2500    32.8->9.1 us    82.0->22.7 ms    184,421 -> 21,925  (73.8->8.8/dev)
>>>     5000    33.0->8.7 us   165.1->43.5 ms    369,401 -> 44,406  (73.9->8.9/dev)
>>>
>>> The reproducer is synthetic: an in-kernel module registers N unbound
>>> platform devices (60 attributes / 4 groups, ~74 kernfs nodes each; no
>>> config space, BARs, or probe) in one timed burst that models SR-IOV
>>> enablement. It runs as a two-kernel A/B from one vanilla 7.1-rc2 tree
>>> (eager baseline vs patched, plus a patched/opt-in-OFF arm), 20 runs per
>>> point. Measures only sysfs-creation slice.
>>
>> I sure hope sr-iov devices are NOT platform devices.  If so, please go
>> fix that up first :)
>>
>>> Deferral removes ~74% of the sysfs-creation time and ~88% of the kernfs
>>> nodes. At 5000 devices that is 121.6 ms removed (165.1 -> 43.5 ms) and
>>> ~325,000 fewer kernfs nodes (369,401 -> 44,406). The table-driven rework
>>> adds a +7..+11% eager-path time overhead to non-opted devices.
>>
>> You're just deferring this work to happen later, so when is that
>> "later"?  And what about multi-threaded probing of the bus, doesn't that
>> speed stuff up too?

The primary goal is move it out of live update hot path. Later is an 
arbitrary point where it doesn't cause CPU downtime of the guest, but merely
causes some VF related operation to take slightly longer. Multi-threaded
probing still hits the kernfs limitation.

>>
>>> Available as a docker image that orchestrates builds in qemu guest and
>>> prints results as a table (x86_64 Linux host with /dev/kvm assumed):
>>>
>>>    docker run --device /dev/kvm ghcr.io/pavsa/linux-lazy-sysfs-vf-bench:7.1-rc2
>>>
>>> Assumptions / Trade-offs
>>> ------------------------
>>> The saving depends on the access pattern of hypervisors, and on no
>>> component walking the whole device tree on the hot path to avoid full
>>> materialization. Both are expected to hold on optimized hypervisors.
>>
>> Hah, as if.  Many userspace tools, once a device is notified, walk the
>> whole tree of it to read the attributes.  You are going to be fighting
>> that problem constantly...
> 
> This series could still benefit Live Update scenarios where the
> initial userspace stack is small and tightly controlled to resume VMs
> as quickly as possible after kexec-ing into the new kernel. It enables
> userspace to decide when to incur the cost of creating these files (or
> never create them).
> 
> But I agree that eliminating the time entirely would be better.
> 

That's the scenario we're targeting.

>>
>>> An actual production hypervisor was observed in local testing to
>>> materialize an additional 5-10% of total nodes on average on the hot
>>> path.
>>>
>>> Overhead: +7..+11% eager-path time overhead applies to every non-opted
>>> device going through device_add() as all struct devices traverse the
>>> walker and can't be gated under a config option.
>>> The lazy opt-in infra can be gated under config option to have zero
>>> impact if disabled.
>>
>> Again, where exactly is the time being spent?  What lock is "hot"?

As noted above, the kernfs lock popped up in analysis - it seems to
be relatively expensive to create nodes at this scale.

>>
>>> Final words
>>> -----------
>>> This is an RFC to first get feedback on the decisions taken at driver
>>> core + kernfs level before engaging maintainers of other subsystems
>>> if we want to pursue this further or pivot to something else.
>>>
>>> The code is at a proof-of-concept quality written with AI assistance
>>> with some known limitations with main target to gather potential
>>> savings and initiate conversation.
>>>
>>> Feedback appreciated for:
>>> - table-driven attribute refactor targeting common device add path /
>>>   suggestions for more efficient/less invasive approach
>>
>> What's wrong with the normal way attributes are defined and allocated?
>> They should all be tables today, and the driver core creating them when
>> needed, no individual driver should ever be doing that on its own.
>>
>> thanks,
>>
>> greg k-h

The attributes themselves are in tables that's true, though decision
making around which attributes to create for what device is not. Having
that in tables also allows them to be materialized later, given that
rules remain static.