[PATCH v9 00/29] Native SCSI Multipath support
Laurence Oberman
loberman at redhat.com
Mon Nov 9 12:19:32 PST 2026
On Sun, 2026-11-08 at 20:47 +0000, John Garry wrote:
> NATIVE SCSI MULTIPATH:
>
> This series introduces native SCSI multipath support. It is intended
> as
> an alternative to dm-mpath.
>
> This support aims to provide a multipath-enabled SCSI block device/
> gendisk.
>
> For a SCSI device to support native multipath, either of the
> following
> conditions must be satisfied:
> a. unique ID in VPD page 83 and ALUA support and scsi_multipath
> modparam
> enabled
> b. unique ID in VPD page 83 and scsi_multipath_always modparam
> enabled
>
> This series relies on reading sdev->access_state to get path
> information.
> This path information would be provided by ALUA. ALUA support which
> does
> not rely on device handlers has already been discussed at
> https://lore.kernel.org/linux-scsi/7755e98f-5619-48ba-bcfc-b64eec930c40@oracle.com/
> and support will be added in the next phase.
>
> New classes of devices are added:
> - scsi_mpath_device
> - scsi_mpath_disk
>
> These are required since a multipath scsi_device has no common scsi
> host.
> An example of the sysfs files and directories for these new classes
> is as
> follows:
>
> $ ls -l /sys/class/scsi_mpath_device/scsi_mpath_device0/
> total 0
> -rw-r--r-- 1 root root 4096 Feb 25 11:59 iopolicy
> drwxr-xr-x 2 root root 0 Feb 25 11:59 multipath
> drwxr-xr-x 2 root root 0 Feb 25 11:59 power
> lrwxrwxrwx 1 root root 0 Feb 25 11:59 subsystem ->
> ../../../../class/scsi_mpath_device
> -rw-r--r-- 1 root root 4096 Feb 25 11:58 uevent
> -r--r--r-- 1 root root 4096 Feb 25 11:59 vpd_id
> $ ls -l /sys/class/scsi_mpath_device/scsi_mpath_device0/multipath/
> total 0
> lrwxrwxrwx 1 root root 0 Feb 25 11:59 8:0:0:0 ->
> ../../../../platform/host8/session1/target8:0:0/8:0:0:0
> lrwxrwxrwx 1 root root 0 Feb 25 11:59 9:0:0:0 ->
> ../../../../platform/host9/session2/target9:0:0/9:0:0:0
> $ cat /sys/class/scsi_mpath_device/scsi_mpath_device0/vpd_id
> naa.600140505200a986f0043c9afa1fd077
> $ cat /sys/class/scsi_mpath_device/scsi_mpath_device0/iopolicy
> numa
> $
>
> $ ls -l /sys/class/scsi_mpath_disk/scsi_mpath_disk0/
> total 0
> drwxr-xr-x 2 root root 0 Feb 25 12:00 power
> drwxr-xr-x 11 root root 0 Feb 25 11:58 sdc
> lrwxrwxrwx 1 root root 0 Feb 25 11:58 subsystem ->
> ../../../../class/scsi_mpath_disk
> -rw-r--r-- 1 root root 4096 Feb 25 11:58 uevent
> $ ls -l /sys/class/scsi_mpath_disk/scsi_mpath_disk0/sdc/multipath/
> total 0
> lrwxrwxrwx 1 root root 0 Feb 25 12:00 sdc:0 ->
> ../../../../../platform/host8/session1/target8:0:0/8:0:0:0/block/sdc:
> 0
> lrwxrwxrwx 1 root root 0 Feb 25 12:00 sdc:1 ->
> ../../../../../platform/host9/session2/target9:0:0/9:0:0:0/block/sdc:
> 1
>
> $ ls -l /dev/sdc
> brw-rw---- 1 root disk 8, 32 Feb 25 11:58 /dev/sdc
>
> The scsi_device and scsi_disk classes otherwise remain unmodified.
> However, the per-path block device is hidden in /dev/. Furthermore,
> multipathed block devices have a new naming scheme, sdX:Y, where
> X is the scsi multipath device index and Y is the path index.
>
> No multipath sg support is added. We still have a per-path sg device.
> Since the SCSI block device is multipath enabled, we can access
> multipathed scsi_ioctl() through that block device.
>
> For failover, we take the approach of cloning bio's and re-submitting
> them
> in full (for failover errors).
>
> LIBMULTIPATH:
>
> This series introduces also libmultipath. It is essentially a
> refactoring
> of NVME multipath support, so we can have a common library to also
> support
> native SCSI multipath.
>
> Much of the code is taken directly from the NVMe multipath code.
> However,
> NVMe specifics are removed. A template structure is provided so the
> driver
> -may provide callbacks for driver specifics, like ANA support for
> NVMe.
>
> Important new structures introduced include:
>
> - mpath_head
> These contain much of the multipath-specific functionality from
> nvme_ns_head, including a pointer to the gendisk structure and
> a path SRCU-based array.
>
> - mpath_device
> This is the per-path structure, and contains much the same
> multipath-specific functionality in nvme_ns
>
> libmultipath provides functionality for path management, path
> selection,
> data path, and failover handling.
>
> Full series also available at
> https://github.com/johnpgarry/linux/tree/scsi-multipath-v7.4-v9
>
> Differences to v8:
> - Rebase
> - Hardening changes advised by sashiko, including "Clear cmd->flags"
> change
>
> Differences to v7:
> - rebase
> - simplify scsi_mpath_clone_end_io() (sashiko)
> - rename bio_list_add_clone() -> bio_list_add_master()
> - rework checks in mpath_alloc_disk_head() (sashiko)
>
> John Garry (29):
> scsi: core: Clear cmd->flags for terminal queuing error
> libmultipath: Add initial framework
> libmultipath: Add basic gendisk support
> libmultipath: Add path selection support
> libmultipath: Add bio handling
> libmultipath: Add support for mpath_device management
> libmultipath: Add delayed removal support
> libmultipath: Add sysfs helpers
> libmultipath: Add support for block device IOCTL
> libmultipath: Add mpath_bdev_getgeo()
> libmultipath: Add mpath_bdev_get_unique_id()
> scsi-multipath: introduce basic SCSI device support
> scsi-multipath: introduce scsi_device head structure
> scsi-multipath: provide sysfs link from to scsi_device
> scsi-multipath: support iopolicy
> scsi-multipath: clone each bio
> scsi-multipath: clear path when device is blocked
> scsi-multipath: revalidate paths upon device unblock
> scsi-multipath: failover handling
> scsi-multipath: provide callbacks for path state
> scsi-multipath: add scsi_mpath_{start,end}_request()
> scsi-multipath: update request for terminal request queuing error
> scsi-multipath: add delayed disk removal support
> scsi: sd: add multipath disk class
> scsi: sd: add multipath disk attr groups
> scsi: sd: support multipath disk
> scsi: sd: add mpath_dev file
> scsi: sd: add mpath_numa_nodes dev attribute
> scsi: sd: add mpath_queue_depth dev attribute
>
> drivers/scsi/Kconfig | 10 +
> drivers/scsi/Makefile | 1 +
> drivers/scsi/scsi.c | 8 +-
> drivers/scsi/scsi_lib.c | 20 +-
> drivers/scsi/scsi_multipath.c | 691 +++++++++++++++++++++++++
> drivers/scsi/scsi_scan.c | 4 +
> drivers/scsi/scsi_sysfs.c | 10 +
> drivers/scsi/sd.c | 611 +++++++++++++++++++++-
> drivers/scsi/sd.h | 3 +
> include/linux/multipath.h | 156 ++++++
> include/scsi/scsi_cmnd.h | 9 +-
> include/scsi/scsi_device.h | 2 +
> include/scsi/scsi_driver.h | 4 +
> include/scsi/scsi_host.h | 4 +
> include/scsi/scsi_multipath.h | 129 +++++
> lib/Kconfig | 6 +
> lib/Makefile | 2 +
> lib/multipath.c | 948
> ++++++++++++++++++++++++++++++++++
> 18 files changed, 2592 insertions(+), 26 deletions(-)
> create mode 100644 drivers/scsi/scsi_multipath.c
> create mode 100644 include/linux/multipath.h
> create mode 100644 include/scsi/scsi_multipath.h
> create mode 100644 lib/multipath.c
>
>
>
On 08/11/2026 20:47, John Garry wrote:
> This series relies on reading sdev->access_state to get path
information.
> This path information would be provided by ALUA. ALUA support which
does
> not rely on device handlers has already been discussed at
>
https://lore.kernel.org/linux-scsi/7755e98f-5619-48ba-bcfc-b64eec930c40@oracle.com/
> and support will be added in the next phase.
Hi John,
I have been testing v9 (your scsi-multipath-v7.4-v9 branch, on
7.4-rc1) on real hardware and found two problems with ALUA arrays,
one in scsi_dh_alua and one in the series itself.
Setup:
- HP P2000 G3 FC array, implicit ALUA, two port groups (one per
controller)
- 2 FC HBAs, 2 target ports each, 30 LUNs with 4 paths per LUN
- scsi_mod.multipath=on, dm-multipath not in use
- Under dm-multipath, the same LUNs show port group 1 at prio 50
(active/optimized) and port group 0 at prio 10
(active/non-optimized)
1. sdev->access_state is wrong after boot (scsi_dh_alua, not this
series)
After boot, every path read active/optimized in
/sys/class/scsi_device/H:C:T:L/device/access_state, while sg_rtpg
reports port group 0 as active/non-optimized. No
"alua: port group XX state ..." message was logged at all.
Since 50759b881e1d ("scsi: device_handler: alua: Call
scsi_device_put() from non-atomic context"), alua_rtpg_queue() only
queues an RTPG if sdev->handler_data is set, but alua_bus_attach()
sets handler_data after alua_initialize() has tried to queue the
initial RTPG. That RTPG is dropped, and access_state stays at its
zeroed default (active/optimized) until something calls
scsi_dh_activate(). dm-multipath hides this through pg_init; native
multipath never calls activate, so the series sees every path as
optimized.
I have sent a separate fix for this to linux-scsi:
https://lore.kernel.org/linux-scsi/20261109201120.1067815-1-loberman@redhat.com/
It is independent of your series, but without it, native multipath
on ALUA arrays has no correct path state.
2. ANO paths are ranked the same as AO paths
With the ALUA state corrected (by writing "activate" to dh_state, or
with the fix above), I/O still went to active/non-optimized paths.
scsi_multipath_sdev_init() sets mpath_device->access_state to
MPATH_STATE_OPTIMIZED and nothing updates it, while
__mpath_find_path(), mpath_round_robin_path() and
mpath_queue_depth_path() rank paths by that field. Only
scsi_mpath_is_optimized()/is_disabled() read sdev->access_state.
With numa, the first path in dev_list wins for every node, and
dev_list order follows sd probe completion, so for about a third of
the LUNs the current path was an ANO path. Example, LUN sdn, reads
completed per path for a 2000 x 1M O_DIRECT dd (:0/:3 AO, :1/:2 ANO
at the time):
sdn:0 0 sdn:1 2000 sdn:2 0 sdn:3 0
Every read went through the non-owning controller. From the code, for
these LUNs the numa recheck of is_optimized() fails on every I/O and
__mpath_find_path() picks the same ANO path again.
With the change below (a template callback so the selectors use the
live state), plus the scsi_dh_alua fix, all 30 LUNs pick an AO
current path at boot, and on LUN sdl (:0/:2 AO, :1/:3 ANO):
numa 2000 0 0 0
round-robin 1000 0 1000 0
queue-depth 18039 0 13961 0 (8 parallel dd, 64k)
No I/O reached the ANO paths under any policy. This is a test patch to
show the problem, not a polished change - feel free to fold it in or
do it differently.
3. Smaller points
- numa with equal NUMA distance uses one path per LUN, so with two AO
paths through two HBAs, half the optimized bandwidth is idle (sdl:2
above under numa). That matches NVMe, but users coming from
dm-multipath service-time will notice. Would round-robin or
queue-depth be a better default for SCSI?
- There are no Documentation/ABI entries for the new sysfs attributes
(scsi_mpath_device vpd_id/iopolicy/multipath, delayed_removal_secs,
mpath_dev, mpath_numa_nodes, mpath_queue_depth), and the
scsi_mod.multipath and multipath_iopolicy parameters are not in
kernel-parameters.txt. I have a draft ABI file if useful.
- There is no userspace tool yet to show heads, paths and ALUA state
(the equivalent of multipath -ll or nvme list-subsys). I used a
small sysfs script for this testing. Ben, is native SCSI multipath
support in multipath-tools planned?
- CONFIG_SCSI_MULTIPATH/CONFIG_LIBMULTIPATH are only enabled in your
branch's x86_64_defconfig, so building from a distro config leaves
them off, and scsi_mod.multipath= is silently ignored. Maybe worth a
note in the cover letter.
I have not tested failover or path loss yet; I will follow up with
that and a Tested-by once done.
Thanks,
Laurence
-- >8 --
From: Laurence Oberman <loberman at redhat.com>
Subject: [PATCH TEST] libmultipath/scsi-multipath: rank paths by live
ALUA access state
mpath_device->access_state is set to MPATH_STATE_OPTIMIZED in
scsi_multipath_sdev_init() and never updated, so the path selectors
rank active/non-optimized paths equal to active/optimized ones.
Add an optional mpath_head_template.access_state() callback, used by
the selectors in place of the cached field when provided, and
implement it for SCSI by mapping the live sdev->access_state.
No change notification is needed: numa rechecks is_optimized() per
I/O and re-runs __mpath_find_path() when the current path is no
longer optimized; round-robin and queue-depth walk the list per I/O.
Signed-off-by: Laurence Oberman <loberman at redhat.com>
---
diff --git a/drivers/scsi/scsi_multipath.c
b/drivers/scsi/scsi_multipath.c
index 58b0be88a..bdae08ecd 100644
--- a/drivers/scsi/scsi_multipath.c
+++ b/drivers/scsi/scsi_multipath.c
@@ -330,6 +330,23 @@ static bool scsi_mpath_is_optimized(struct
mpath_device *mpath_device)
return access_state == SCSI_ACCESS_STATE_OPTIMAL;
}
+static enum mpath_access_state scsi_mpath_access_state(
+ struct mpath_device *mpath_device)
+{
+ struct scsi_mpath_device *scsi_mpath_dev =
+ to_scsi_mpath_device(mpath_device);
+ struct scsi_device *sdev = scsi_mpath_dev->sdev;
+
+ switch (READ_ONCE(sdev->access_state) &
SCSI_ACCESS_STATE_MASK) {
+ case SCSI_ACCESS_STATE_OPTIMAL:
+ return MPATH_STATE_OPTIMIZED;
+ case SCSI_ACCESS_STATE_ACTIVE:
+ return MPATH_STATE_NONOPTIMIZED;
+ default:
+ return MPATH_STATE_OTHER;
+ }
+}
+
static bool scsi_mpath_available_path(struct mpath_device
*mpath_device)
{
struct scsi_mpath_device *scsi_mpath_dev =
@@ -374,6 +391,7 @@ static struct mpath_head_template smpdt = {
.remove_head = scsi_mpath_remove_head_work,
.is_disabled = scsi_mpath_is_disabled,
.is_optimized = scsi_mpath_is_optimized,
+ .access_state = scsi_mpath_access_state,
.available_path = scsi_mpath_available_path,
.clone_bio = scsi_mpath_clone_bio,
};
diff --git a/include/linux/multipath.h b/include/linux/multipath.h
index 7a3708ff3..b47785804 100644
--- a/include/linux/multipath.h
+++ b/include/linux/multipath.h
@@ -39,6 +39,7 @@ struct mpath_head_template {
void (*remove_head)(struct mpath_head *);
bool (*is_disabled)(struct mpath_device *);
bool (*is_optimized)(struct mpath_device *);
+ enum mpath_access_state (*access_state)(struct mpath_device
*);
struct bio *(*clone_bio)(struct bio *);
const struct attribute_group **device_groups;
};
diff --git a/lib/multipath.c b/lib/multipath.c
index b03080afb..3ddb7296a 100644
--- a/lib/multipath.c
+++ b/lib/multipath.c
@@ -142,6 +142,19 @@ static bool mpath_path_is_disabled(struct
mpath_head *mpath_head,
return mpath_head->mpdt->is_disabled(mpath_device);
}
+/*
+ * Use the driver's live view of the path access state if it provides
one,
+ * otherwise fall back to the cached mpath_device->access_state.
+ */
+static enum mpath_access_state mpath_path_access_state(
+ struct mpath_head *mpath_head,
+ struct mpath_device *mpath_device)
+{
+ if (mpath_head->mpdt->access_state)
+ return mpath_head->mpdt->access_state(mpath_device);
+ return mpath_device->access_state;
+}
+
static struct mpath_device *__mpath_find_path(struct mpath_head
*mpath_head,
int node)
{
@@ -161,7 +174,7 @@ static struct mpath_device
*__mpath_find_path(struct mpath_head *mpath_head,
else
distance = LOCAL_DISTANCE;
- switch(mpath_device->access_state) {
+ switch (mpath_path_access_state(mpath_head,
mpath_device)) {
case MPATH_STATE_OPTIMIZED:
if (distance < found_distance) {
found_distance = distance;
@@ -206,7 +219,7 @@ static struct mpath_device *mpath_round_robin_path(
{
struct mpath_device *mpath_device, *found = NULL;
int node = numa_node_id();
- enum mpath_access_state access_state_old;
+ enum mpath_access_state access_state_old, state;
struct mpath_device *old =
srcu_dereference(mpath_head-
>current_path[node],
&mpath_head->srcu);
@@ -226,11 +239,12 @@ static struct mpath_device
*mpath_round_robin_path(
if (mpath_path_is_disabled(mpath_head, mpath_device))
continue;
- if (mpath_device->access_state ==
MPATH_STATE_OPTIMIZED) {
+ state = mpath_path_access_state(mpath_head,
mpath_device);
+ if (state == MPATH_STATE_OPTIMIZED) {
found = mpath_device;
goto out;
}
- if (mpath_device->access_state ==
MPATH_STATE_NONOPTIMIZED)
+ if (state == MPATH_STATE_NONOPTIMIZED)
found = mpath_device;
}
@@ -240,7 +254,7 @@ static struct mpath_device *mpath_round_robin_path(
* - no other optimized path found and current is optimized,
* - no other usable path found and current is usable.
*/
- access_state_old = old->access_state;
+ access_state_old = mpath_path_access_state(mpath_head, old);
if (!mpath_path_is_disabled(mpath_head, old) &&
(access_state_old == MPATH_STATE_OPTIMIZED ||
(!found && access_state_old == MPATH_STATE_NONOPTIMIZED)))
@@ -270,7 +284,7 @@ static struct mpath_device *mpath_queue_depth_path(
depth = atomic_read(mpath_device->nr_active);
- switch (mpath_device->access_state) {
+ switch (mpath_path_access_state(mpath_head,
mpath_device)) {
case MPATH_STATE_OPTIMIZED:
if (depth < min_depth_opt) {
min_depth_opt = depth;
More information about the Linux-nvme
mailing list