From hch at lst.de  Sun Oct  1 00:42:03 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 1 Oct 2017 09:42:03 +0200
Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
References: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
Message-ID: <20171001074203.GA11115@lst.de>

This looks very convoluted, mostly because the existing code is
doing weird things.  For one thing what is sq_dma_addr currently
is not a DMA adddress - we either need the resource address
for the ioremap, but we don't need to stash that away, and second
the one programmed into the controller should be a pci_bus_addr_t.

Second we already have a nice PCI-layer helper called pci_bus_address
to get the bus address for us and we should use it.

Something like the patch below should solve the issue:

---
>From b78f4164881125c4fecfdb87878d0120b2177c53 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <hch at lst.de>
Date: Sun, 1 Oct 2017 09:37:35 +0200
Subject: nvme-pci: Use PCI bus address for data/queues in CMB

Currently, NVMe PCI host driver is programming CMB dma address as
I/O SQs addresses. This results in failures on systems where 1:1
outbound mapping is not used (example Broadcom iProc SOCs) because
CMB BAR will be progammed with PCI bus address but NVMe PCI EP will
try to access CMB using dma address.

To have CMB working on systems without 1:1 outbound mapping, we
program PCI bus address for I/O SQs instead of dma address. This
approach will work on systems with/without 1:1 outbound mapping.

Based on a report and previous patch from Abhishek Shah.

Fixes: 8ffaadf7 ("NVMe: Use CMB for the IO SQes if available")
Cc: stable at vger.kernel.org
Reported-by: Abhishek Shah <abhishek.shah at broadcom.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/pci.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cb73bc8cad3b..3f5a04c586ce 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -94,7 +94,7 @@ struct nvme_dev {
 	struct mutex shutdown_lock;
 	bool subsystem;
 	void __iomem *cmb;
-	dma_addr_t cmb_dma_addr;
+	pci_bus_addr_t cmb_bus_addr;
 	u64 cmb_size;
 	u32 cmbsz;
 	u32 cmbloc;
@@ -1226,7 +1226,7 @@ static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
 	if (qid && dev->cmb && use_cmb_sqes && NVME_CMB_SQS(dev->cmbsz)) {
 		unsigned offset = (qid - 1) * roundup(SQ_SIZE(depth),
 						      dev->ctrl.page_size);
-		nvmeq->sq_dma_addr = dev->cmb_dma_addr + offset;
+		nvmeq->sq_dma_addr = dev->cmb_bus_addr + offset;
 		nvmeq->sq_cmds_io = dev->cmb + offset;
 	} else {
 		nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),
@@ -1527,7 +1527,7 @@ static void __iomem *nvme_map_cmb(struct nvme_dev *dev)
 	resource_size_t bar_size;
 	struct pci_dev *pdev = to_pci_dev(dev->dev);
 	void __iomem *cmb;
-	dma_addr_t dma_addr;
+	int bar;
 
 	dev->cmbsz = readl(dev->bar + NVME_REG_CMBSZ);
 	if (!(NVME_CMB_SZ(dev->cmbsz)))
@@ -1540,7 +1540,8 @@ static void __iomem *nvme_map_cmb(struct nvme_dev *dev)
 	szu = (u64)1 << (12 + 4 * NVME_CMB_SZU(dev->cmbsz));
 	size = szu * NVME_CMB_SZ(dev->cmbsz);
 	offset = szu * NVME_CMB_OFST(dev->cmbloc);
-	bar_size = pci_resource_len(pdev, NVME_CMB_BIR(dev->cmbloc));
+	bar = NVME_CMB_BIR(dev->cmbloc);
+	bar_size = pci_resource_len(pdev, bar);
 
 	if (offset > bar_size)
 		return NULL;
@@ -1553,12 +1554,11 @@ static void __iomem *nvme_map_cmb(struct nvme_dev *dev)
 	if (size > bar_size - offset)
 		size = bar_size - offset;
 
-	dma_addr = pci_resource_start(pdev, NVME_CMB_BIR(dev->cmbloc)) + offset;
-	cmb = ioremap_wc(dma_addr, size);
+	cmb = ioremap_wc(pci_resource_start(pdev, bar) + offset, size);
 	if (!cmb)
 		return NULL;
 
-	dev->cmb_dma_addr = dma_addr;
+	dev->cmb_bus_addr = pci_bus_address(pdev, bar) + offset;
 	dev->cmb_size = size;
 	return cmb;
 }
-- 
2.14.1



From hch at lst.de  Sun Oct  1 00:58:31 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 1 Oct 2017 09:58:31 +0200
Subject: [PATCH 1/3] nvme: fix visibility of "uuid" ns attribute
In-Reply-To: <20170928193323.24545-1-mwilck@suse.com>
References: <20170928193323.24545-1-mwilck@suse.com>
Message-ID: <20171001075831.GC11554@lst.de>

Thanks,

applied to nvme-4.14.


From hch at lst.de  Sun Oct  1 01:00:03 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 1 Oct 2017 10:00:03 +0200
Subject: [PATCH 1/2] block: genhd: add device_add_disk_with_groups
In-Reply-To: <20170928193637.24707-1-mwilck@suse.com>
References: <20170928193637.24707-1-mwilck@suse.com>
Message-ID: <20171001080003.GD11554@lst.de>

While this looks okay-ish to me I really don't want people confused
with three variants of add_disk, we really need to consolidate
our helpers there a bit..


From hch at lst.de  Sun Oct  1 01:07:03 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 1 Oct 2017 10:07:03 +0200
Subject: [PATCH 8/9] nvme: implement multipath access to nvme subsystems
In-Reply-To: <mqdk20ghta5.fsf@linux-x5ow.site>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-9-hch@lst.de>
 <20170928155315.GK8463@localhost.localdomain>
 <mqdk20ghta5.fsf@linux-x5ow.site>
Message-ID: <20171001080703.GA11895@lst.de>

On Sat, Sep 30, 2017 at 09:37:38PM +0200, Johannes Thumshirn wrote:
> We discussed (Hannes, Christoph and me) this as well offline this week
> and it should eithr be possible with some udev magic (fake a
> /dev/mapper/WWID symlink) or a shim device-mapper target that prevents
> dm-mpath from attaching to the underlying block devices and creates a
> real /dev/mapper/WWID device node (Christoph especially dislikes
> this). None of them are pretty, but probably all we can do so far.

I think the former is the way to go.

That being said I think you really need to kick the installers peoples
ass - at least for local bus devices there is no reason every to not
just support any node in /dev/.  We can triviall go from that to the
modules required in sysfs.  Anything networkish will need additional
work for setup that is more than bus probing, but we really should
not work around people being lazy and/or stupid too much.


From hch at lst.de  Sun Oct  1 01:07:32 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 1 Oct 2017 10:07:32 +0200
Subject: [PATCH 8/9] nvme: implement multipath access to nvme subsystems
In-Reply-To: <CADL4p-rbv8eH0UYjPkA2oTNafZgv16k0rmJEcj2=jz7yyYf8KA@mail.gmail.com>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-9-hch@lst.de>
 <20170928155315.GK8463@localhost.localdomain>
 <CADL4p-rbv8eH0UYjPkA2oTNafZgv16k0rmJEcj2=jz7yyYf8KA@mail.gmail.com>
Message-ID: <20171001080732.GB11895@lst.de>

On Fri, Sep 29, 2017 at 10:21:53PM +0800, Tony Yang wrote:
> Hi, All
> 
>      Because my environment requirements, the kernel must use 4.8.17,
> I would like to ask, how to use the kernel 4.8.17 nvme multi-path?
> Because I see support for multi-path versions are above 4.13

In that case we simply can't help you.


From hch at infradead.org  Sun Oct  1 01:39:32 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Sun, 1 Oct 2017 01:39:32 -0700
Subject: [PATCH] nvmet_fc: Reduce work_q count
In-Reply-To: <e86009d5-2ca6-8de3-d142-781724bf00d8@gmail.com>
References: <20170927040958.5981-1-jsmart2021@gmail.com>
 <90b9711d-e279-443d-1693-dad24524cc74@grimberg.me>
 <e86009d5-2ca6-8de3-d142-781724bf00d8@gmail.com>
Message-ID: <20171001083932.GA878@infradead.org>

On Sat, Sep 30, 2017 at 11:32:39AM -0700, James Smart wrote:
> Because we're still trying to get cpu affinity on the connection resources
> that will be shared by the io on the same queue.

And that's exactly what plain queue_work will give you.  If it doesn't
do exactly what your open code logic does it has a bug that needs to
be fixed.


From hch at infradead.org  Sun Oct  1 01:47:00 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Sun, 1 Oct 2017 01:47:00 -0700
Subject: [PATCH] nvme_fc: correct io timeout behavior
In-Reply-To: <20170927041317.6575-1-jsmart2021@gmail.com>
References: <20170927041317.6575-1-jsmart2021@gmail.com>
Message-ID: <20171001084700.GB878@infradead.org>

> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -1374,7 +1374,7 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>  				sizeof(op->rsp_iu), DMA_FROM_DEVICE);
>  
>  	if (atomic_read(&op->state) == FCPOP_STATE_ABORTED)
> -		status = cpu_to_le16((NVME_SC_ABORT_REQ | NVME_SC_DNR) << 1);
> +		status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);

So the DNR bit goes away, ok.

> @@ -1445,15 +1445,22 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>  		goto check_error;
>  	}
>  
> +	if (unlikely(!status && op->flags & FCOP_FLAGS_TERMIO))
> +		status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);

We grow another aborted case.  And btw, we really should not use
NVME_SC_ABORT_REQ here because the command never reched the controller
to actually get a nvme abort.  We might have to live with this for
now, but can you please review might addition of new status codes
for these host-internal cases in the latest ANA draft?

> +	/*
> +	 * Force failures of commands if we're killing the controller
> +	 * or have an error on a command used to create an new association
> +	 */
> +	if (status && (blk_queue_dying(rq->q) ||
> +			ctrl->ctrl.state == NVME_CTRL_NEW ||
> +			ctrl->ctrl.state == NVME_CTRL_RECONNECTING))
> +		status |= cpu_to_le16(NVME_SC_DNR << 1);

And here we add back the dnr bit based on a few conditions.
As a nitpick I think the way the conditionals are nested isn't very
readable, can we split the first line up, please?

	 if (status &&
	     (blk_queue_dying(rq->q) ||
	      ctrl->ctrl.state == NVME_CTRL_NEW ||
	      ctrl->ctrl.state == NVME_CTRL_RECONNECTING))
		status |= cpu_to_le16(NVME_SC_DNR << 1);

>  	complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
> +	if (!complete_rq)
>  		nvme_end_request(rq, status, result);
> +	else
>  		__nvme_fc_final_op_cleanup(rq);

avoid the unneeded invert and local variable:

	if (unlikely(__nvme_fc_fcpop_chk_teardowns(ctrl, op)))
		__nvme_fc_final_op_cleanup(rq);
	else
		nvme_end_request(rq, status, result);

and while we're at it: the AEN special case above assigns complete_rq,
but that value will never be read, so the complete_rq can be removed
entirely.

Can you split the reset handler bits into a separate patch with
a separate explanation?


From bhumirks at gmail.com  Sun Oct  1 12:14:38 2017
From: bhumirks at gmail.com (Bhumika Goyal)
Date: Mon,  2 Oct 2017 00:44:38 +0530
Subject: [PATCH] nvme: make config_item_type __ro_after_init
Message-ID: <1506885278-9234-1-git-send-email-bhumirks@gmail.com>

Make these __ro_after_init as they are only stored as a reference in the
ci_type field of a config_item structure during the init phase in the
function config_group_init_type_name. None of the fields of these
structures are modified after the init phase, so they can be
__ro_after_init.

Structures found using Coccinelle and changes done by hand.

Signed-off-by: Bhumika Goyal <bhumirks at gmail.com>
---
 drivers/nvme/target/configfs.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index b6aeb1d..075d9b5 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -767,7 +767,7 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	.make_group		= nvmet_subsys_make,
 };
 
-static struct config_item_type nvmet_subsystems_type = {
+static struct config_item_type nvmet_subsystems_type __ro_after_init = {
 	.ct_group_ops		= &nvmet_subsystems_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -921,7 +921,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 	.make_group		= nvmet_ports_make,
 };
 
-static struct config_item_type nvmet_ports_type = {
+static struct config_item_type nvmet_ports_type __ro_after_init = {
 	.ct_group_ops		= &nvmet_ports_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -963,7 +963,7 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
 	.make_group		= nvmet_hosts_make_group,
 };
 
-static struct config_item_type nvmet_hosts_type = {
+static struct config_item_type nvmet_hosts_type __ro_after_init = {
 	.ct_group_ops		= &nvmet_hosts_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
-- 
1.9.1



From hare at suse.de  Sun Oct  1 23:19:30 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 2 Oct 2017 08:19:30 +0200
Subject: [PATCH 8/9] nvme: implement multipath access to nvme subsystems
In-Reply-To: <mqdk20ghta5.fsf@linux-x5ow.site>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-9-hch@lst.de>
 <20170928155315.GK8463@localhost.localdomain>
 <mqdk20ghta5.fsf@linux-x5ow.site>
Message-ID: <c6fe82f1-558f-50ee-e053-d55f4ec54801@suse.de>

On 09/30/2017 09:37 PM, Johannes Thumshirn wrote:
> 
> [+Cc Hannes ]
> 
> Keith Busch <keith.busch at intel.com> writes:
> 
>> On Mon, Sep 25, 2017 at 03:40:30PM +0200, Christoph Hellwig wrote:
>>> The new block devices nodes for multipath access will show up as
>>>
>>> 	/dev/nvm-subXnZ
>>
>> Just thinking ahead ... Once this goes in, someone will want to boot their
>> OS from a multipath target. It was a pain getting installers to recognize
>> /dev/nvmeXnY as an install destination. I'm not sure if installers have
>> gotten any better in the last 5 years about recognizing new block names.
> 
> We discussed (Hannes, Christoph and me) this as well offline this week
> and it should eithr be possible with some udev magic (fake a
> /dev/mapper/WWID symlink) or a shim device-mapper target that prevents
> dm-mpath from attaching to the underlying block devices and creates a
> real /dev/mapper/WWID device node (Christoph especially dislikes
> this). None of them are pretty, but probably all we can do so far.
> 
There are several issues here.
The one is the device-naming issue; I'm heavily against the nvm-subX
thing. Why don't we just name them

/dev/nvmsXnZ

which nicely aligns with the existing /dev/nvmeXnZ naming scheme.

The other issue is how to enable multipath booting. Painful experience
tells us we _need_ to be able to figure out if a device is multipath
capable without having to invoke any external tools, ie the device
itself need to carry some marker 'this device is multipath capable'.
In NVMe speak this means we have to export the CMIC bit from the
Identify controller data structure to somewhere in sysfs.

Then it's easy to setup a udev rule which sets 'SYSTEMD_READY=0'
whenever this setting is detected, and the entire udev mechanism will
leave it alone.

Once this is done we have the paths nicely separated in udev, and then
setting up the symlinks pointing to the correct device is trivial.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From bhumirks at gmail.com  Mon Oct  2 01:18:38 2017
From: bhumirks at gmail.com (Bhumika Goyal)
Date: Mon, 2 Oct 2017 13:48:38 +0530
Subject: [PATCH] nvme: make config_item_type __ro_after_init
In-Reply-To: <1506885278-9234-1-git-send-email-bhumirks@gmail.com>
References: <1506885278-9234-1-git-send-email-bhumirks@gmail.com>
Message-ID: <CAOH+1jFW5b2C05kP9FHHT_529kCy4cJ2z6QF9XyUHYhMixXPfQ@mail.gmail.com>

On Mon, Oct 2, 2017 at 12:44 AM, Bhumika Goyal <bhumirks at gmail.com> wrote:
> Make these __ro_after_init as they are only stored as a reference in the
> ci_type field of a config_item structure during the init phase in the
> function config_group_init_type_name. None of the fields of these
> structures are modified after the init phase, so they can be
> __ro_after_init.
>
> Structures found using Coccinelle and changes done by hand.
>

Hello all,

This patch should be dropped as the structures can also be made const
instead of __ro_after_init. I will send a patch making them const.

Thanks,
Bhumika

> Signed-off-by: Bhumika Goyal <bhumirks at gmail.com>
> ---
>  drivers/nvme/target/configfs.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index b6aeb1d..075d9b5 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -767,7 +767,7 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
>         .make_group             = nvmet_subsys_make,
>  };
>
> -static struct config_item_type nvmet_subsystems_type = {
> +static struct config_item_type nvmet_subsystems_type __ro_after_init = {
>         .ct_group_ops           = &nvmet_subsystems_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -921,7 +921,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
>         .make_group             = nvmet_ports_make,
>  };
>
> -static struct config_item_type nvmet_ports_type = {
> +static struct config_item_type nvmet_ports_type __ro_after_init = {
>         .ct_group_ops           = &nvmet_ports_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -963,7 +963,7 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
>         .make_group             = nvmet_hosts_make_group,
>  };
>
> -static struct config_item_type nvmet_hosts_type = {
> +static struct config_item_type nvmet_hosts_type __ro_after_init = {
>         .ct_group_ops           = &nvmet_hosts_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> --
> 1.9.1
>


From sagi at grimberg.me  Mon Oct  2 05:51:43 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 2 Oct 2017 14:51:43 +0200
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
Message-ID: <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>

Hi,

>>> Panic after connection with below commits, detailed log here: 
>>> https://pastebin.com/7z0XSGSd
>>> 31fdf18???? nvme-rdma: reuse configure/destroy_admin_queue
>>> 3f02fff?????? nvme-rdma: don't free tagset on resets
>>> 18398af??? nvme-rdma: disable the controller on resets
>>> b28a308?? nvme-rdma: move tagset allocation to a dedicated routine
>>>
>>> good??? 34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl
>>
>> Is that a reproducible panic? I'm not seeing this at all.
>>
> 
> Yes, I can reproduce every time. And the target side kernel version is 
> 4.14.0-rc1 during the panic occurred.
> 
>> Can you run gdb on nvme-rdma.ko
>> $ l *(nvme_rdma_create_ctrl+0x37d)
>>
> [root at rdma-virt-01 linux ((31fdf18...))]$ gdb 
> /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.? Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from 
> /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko...done. 
> 
> (gdb) l *(nvme_rdma_create_ctrl+0x37d)
> 0x297d is in nvme_rdma_create_ctrl (drivers/nvme/host/rdma.c:656).
> 651??????? struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
> 652??????? struct blk_mq_tag_set *set = admin ?
> 653??????????????? &ctrl->admin_tag_set : &ctrl->tag_set;
> 654
> 655??????? blk_mq_free_tag_set(set);
> 656??????? nvme_rdma_dev_put(ctrl->device);
> 657??? }
> 658
> 659??? static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct 
> nvme_ctrl *nctrl,
> 660??????????? bool admin)
> (gdb)

Lets take this one step at a time, starting with this issue.

First, there is a reason why a simple create_ctrl fails, can we isolate
exactly which call fails? Was something else going on that might have
made the simple create_ctrl fail?

We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
messages but we do see "creating I/O queues" which means that we either
failed at IO tagset allocation or initializing connect_q.

We have a missing error code assignment so can you try the following patch:
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 58983000964b..98dd51e630bd 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -765,8 +765,10 @@ static int nvme_rdma_configure_admin_queue(struct 
nvme_rdma_ctrl *ctrl,

         if (new) {
                 ctrl->ctrl.admin_tagset = 
nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
-               if (IS_ERR(ctrl->ctrl.admin_tagset))
+               if (IS_ERR(ctrl->ctrl.admin_tagset)) {
+                       error = PTR_ERR(ctrl->ctrl.admin_tagset);
                         goto out_free_queue;
+               }

                 ctrl->ctrl.admin_q = 
blk_mq_init_queue(&ctrl->admin_tag_set);
                 if (IS_ERR(ctrl->ctrl.admin_q)) {
@@ -846,8 +848,10 @@ static int nvme_rdma_configure_io_queues(struct 
nvme_rdma_ctrl *ctrl, bool new)

         if (new) {
                 ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, 
false);
-               if (IS_ERR(ctrl->ctrl.tagset))
+               if (IS_ERR(ctrl->ctrl.tagset)) {
+                       ret = PTR_ERR(ctrl->ctrl.tagset);
                         goto out_free_io_queues;
+               }

                 ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set);
                 if (IS_ERR(ctrl->ctrl.connect_q)) {
--

Also, can you add the following debug messages to find out what failed?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 58983000964b..e46475100eea 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -676,6 +676,12 @@ static void nvme_rdma_free_tagset(struct nvme_ctrl 
*nctrl, bool admin)
         struct blk_mq_tag_set *set = admin ?
                         &ctrl->admin_tag_set : &ctrl->tag_set;

+       if (set == &ctrl->tag_set) {
+               pr_err("%s: freeing IO tagset\n", __func__);
+       } else {
+               pr_err("%s: freeing ADMIN tagset\n", __func__);
+       }
+
         blk_mq_free_tag_set(set);
         nvme_rdma_dev_put(ctrl->device);
  }
--


From hare at suse.de  Mon Oct  2 06:55:57 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:57 +0200
Subject: [PATCH 4/6] nvme: display 'NMIC' namespace attribute
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-5-git-send-email-hare@suse.de>

From: Johannes Thumshirn <jthumshirn at suse.de>

To figure out if a namespace is shared via multipath we need to expose
the 'NMIC' value to sysfs.

Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 12 ++++++++++++
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 13 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 8924f48..955fedf 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2292,6 +2292,16 @@ static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
 		return disk->private_data;
 }
 
+static ssize_t nmic_show(struct device *dev, struct device_attribute *attr,
+			 char *buf)
+{
+	struct nvme_ns_head *head = dev_to_ns_head(dev);
+	struct nvme_ns_ids *ids = &head->ids;
+
+	return sprintf(buf, "0x%x\n", ids->nmic);
+}
+static DEVICE_ATTR(nmic, S_IRUGO, nmic_show, NULL);
+
 static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 		char *buf)
 {
@@ -2367,6 +2377,7 @@ static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
 	&dev_attr_nguid.attr,
 	&dev_attr_eui.attr,
 	&dev_attr_nsid.attr,
+	&dev_attr_nmic.attr,
 	NULL,
 };
 
@@ -2663,6 +2674,7 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 	kref_init(&head->ref);
 
 	nvme_report_ns_ids(ctrl, nsid, id, &head->ids);
+	head->ids.nmic = id->nmic;
 
 	ret = __nvme_check_ids(ctrl->subsys, head);
 	if (ret) {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index a8a79bc..1f195ff 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -222,6 +222,7 @@ struct nvme_ns_ids {
 	u8	eui64[8];
 	u8	nguid[16];
 	uuid_t	uuid;
+	u8 nmic;
 };
 
 /*
-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:56 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:56 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-4-git-send-email-hare@suse.de>

When setting up a multipath device we need to claim the underlying
block devices to avoid other systems and/or programs to access it.
And we should be using the standard holders/slave sysfs relationship
instead of the hand-crafted 'mpath' links.

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 34 ++++++++++++++++++++++++----------
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index f52f0ab..8924f48 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2805,6 +2805,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	char disk_name[DISK_NAME_LEN];
 	int node = dev_to_node(ctrl->dev);
 	bool new = true;
+	static char *_nvme_claim_ptr = "NVMe shared namespace path";
 
 	ns = kzalloc_node(sizeof(*ns), GFP_KERNEL, node);
 	if (!ns)
@@ -2885,15 +2886,26 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 				ns->head->disk->disk_name);
 	}
 
-	if (sysfs_create_link(&disk_to_dev(ns->disk)->kobj,
-			&disk_to_dev(ns->head->disk)->kobj, "mpath"))
-		pr_warn("%s: failed to create sysfs link to mpath device\n",
+	ns->bdev = bdget_disk(ns->disk, 0);
+	if (!ns->bdev) {
+		pr_warn("%s: failed to get bdev\n", ns->disk->disk_name);
+		return;
+	}
+	if (blkdev_get(ns->bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL,
+		       _nvme_claim_ptr) < 0) {
+		pr_warn("%s: failed to claim bdev\n",
 			ns->disk->disk_name);
-	if (sysfs_create_link(&disk_to_dev(ns->head->disk)->kobj,
-			&disk_to_dev(ns->disk)->kobj, ns->disk->disk_name))
-		pr_warn("%s: failed to create sysfs link from mpath device\n",
+		bdput(ns->bdev);
+		ns->bdev = NULL;
+		nvme_put_ns_head(ns->head);
+		return;
+	}
+	if (bd_link_disk_holder(ns->bdev, ns->head->disk)) {
+		pr_warn("%s: failed to create sysfs link to mpath device\n",
 			ns->disk->disk_name);
-
+		blkdev_put(ns->bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL);
+		ns->bdev = NULL;
+	}
 	return;
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
@@ -2921,11 +2933,13 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 			blk_integrity_unregister(ns->disk);
 		sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
 					&nvme_ns_id_attr_group);
-		sysfs_remove_link(&disk_to_dev(ns->disk)->kobj, "mpath");
-		sysfs_remove_link(&disk_to_dev(ns->head->disk)->kobj,
-				ns->disk->disk_name);
 		if (ns->ndev)
 			nvme_nvm_unregister_sysfs(ns);
+		if (ns->bdev) {
+			bd_unlink_disk_holder(ns->bdev, ns->head->disk);
+			blkdev_put(ns->bdev,
+				   FMODE_READ | FMODE_WRITE | FMODE_EXCL);
+		}
 		del_gendisk(ns->disk);
 		blk_cleanup_queue(ns->queue);
 	}
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 10fffbc..a8a79bc 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -252,6 +252,7 @@ struct nvme_ns {
 	struct nvme_ctrl *ctrl;
 	struct request_queue *queue;
 	struct gendisk *disk;
+	struct block_device *bdev;
 	struct list_head siblings;
 	struct nvm_dev *ndev;
 	struct kref kref;
-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:55 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:55 +0200
Subject: [PATCH 2/6] nvme: use 'nvmsXnZ' instead of 'nvm-subXnZ'
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-3-git-send-email-hare@suse.de>

Align with overall nvme device naming scheme.

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index b1e61c9..f52f0ab 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2689,7 +2689,7 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 	head->disk->private_data = head;
 	head->disk->queue = q;
 	head->disk->flags = GENHD_FL_EXT_DEVT;
-	sprintf(head->disk->disk_name, "nvm-sub%dn%d",
+	sprintf(head->disk->disk_name, "nvms%dn%d",
 			ctrl->subsys->instance, nsid);
 	list_add_tail(&head->entry, &ctrl->subsys->nsheads);
 	return head;
-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:53 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:53 +0200
Subject: [PATCH 0/6] nvme: sanitize multipathing
Message-ID: <1506952559-1588-1-git-send-email-hare@suse.de>

Hi Christoph,

here's a patchset to sanitize nvme multipath handling, following
the discusion we've had at ALPSS.
Here I've settled for the '/dev/nvmsXnZ' device nodes (which you
said I could do :-), and added the necessary bits and piece to
get it registered nicely with sysfs.
We've also found an issue when retrying I/O after path failures;
we probably should be checking if this is multipath I/O before
deciding on retry or not.

Patches are relative to your nvme-mpath branch.

As usual, comments and reviews are welcome.

Hannes Reinecke (5):
  nvme: display 'CMIC' controller attribute
  nvme: use 'nvmsXnZ' instead of 'nvm-subXnZ'
  nvme: claim block devices
  nvme: Export subsystems to /sys/class/nvme-subsys
  nvme: ignore retries for multipath devices

Johannes Thumshirn (1):
  nvme: display 'NMIC' namespace attribute

 drivers/nvme/host/core.c | 99 +++++++++++++++++++++++++++++++++++++-----------
 drivers/nvme/host/nvme.h |  5 ++-
 2 files changed, 80 insertions(+), 24 deletions(-)

-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:54 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:54 +0200
Subject: [PATCH 1/6] nvme: display 'CMIC' controller attribute
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-2-git-send-email-hare@suse.de>

To figure out if a controller is multipath capable (ie if the
controller supports namespace sharing) we need to expose the CMIC
value to sysfs.

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 12 ++++++++++++
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 13 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c296f27..b1e61c9 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2061,6 +2061,7 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 	atomic_set(&ctrl->abort_limit, id->acl + 1);
 	ctrl->vwc = id->vwc;
 	ctrl->cntlid = le16_to_cpup(&id->cntlid);
+	ctrl->cmic = id->cmic;
 	if (id->mdts)
 		max_hw_sectors = 1 << (id->mdts + page_shift - 9);
 	else
@@ -2486,6 +2487,16 @@ static ssize_t nvme_sysfs_show_address(struct device *dev,
 }
 static DEVICE_ATTR(address, S_IRUGO, nvme_sysfs_show_address, NULL);
 
+static ssize_t nvme_sysfs_show_cmic(struct device *dev,
+				    struct device_attribute *attr,
+				    char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return snprintf(buf, PAGE_SIZE, "%d\n", ctrl->cmic);
+}
+static DEVICE_ATTR(cmic, S_IRUGO, nvme_sysfs_show_cmic, NULL);
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -2498,6 +2509,7 @@ static ssize_t nvme_sysfs_show_address(struct device *dev,
 	&dev_attr_subsysnqn.attr,
 	&dev_attr_address.attr,
 	&dev_attr_state.attr,
+	&dev_attr_cmic.attr,
 	NULL
 };
 
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index febc21d..10fffbc 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -170,6 +170,7 @@ struct nvme_ctrl {
 	u16 kas;
 	u8 npss;
 	u8 apsta;
+	u8 cmic;
 	unsigned int shutdown_timeout;
 	unsigned int kato;
 	bool subsystem;
-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:59 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:59 +0200
Subject: [PATCH 6/6] nvme: ignore retries for multipath devices
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-7-git-send-email-hare@suse.de>

For multipath devices we should switch paths after the retries
are exhausted, and not return an error.
IE we should first retry on the current path, and then switch
paths once the retries are exhausted.
Once all paths are down and all retries are exhausted we will
still retry the command, but then it'll be handled with the
all paths down logic in nvme_make_request().

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1ef11ca..ac7676a 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -201,7 +201,8 @@ static inline bool nvme_req_needs_retry(struct request *req)
 		return false;
 	if (nvme_req(req)->status & NVME_SC_DNR)
 		return false;
-	if (nvme_req(req)->retries >= nvme_max_retries)
+	if (nvme_req(req)->retries >= nvme_max_retries &&
+	    !(req->cmd_flags & REQ_NVME_MPATH))
 		return false;
 	return true;
 }
-- 
1.8.5.6



From hare at suse.de  Mon Oct  2 06:55:58 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  2 Oct 2017 15:55:58 +0200
Subject: [PATCH 5/6] nvme: Export subsystems to /sys/class/nvme-subsys
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <1506952559-1588-6-git-send-email-hare@suse.de>

Create a class for each subsystem to create a 'device' link
for each subsystem device.

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 36 +++++++++++++++++++++++++-----------
 drivers/nvme/host/nvme.h |  2 +-
 2 files changed, 26 insertions(+), 12 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 955fedf..1ef11ca 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -79,6 +79,7 @@
 static DEFINE_SPINLOCK(dev_list_lock);
 
 static struct class *nvme_class;
+static struct class *nvme_subsys_class;
 static const struct attribute_group nvme_ns_id_attr_group;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
@@ -1905,10 +1906,10 @@ static void __nvme_free_subsystem(struct nvme_subsystem *subsys)
 	kfree(subsys);
 }
 
-static void nvme_free_subsystem(struct kref *ref)
+static void nvme_free_subsystem(struct device *dev)
 {
 	struct nvme_subsystem *subsys =
-			container_of(ref, struct nvme_subsystem, ref);
+			container_of(dev, struct nvme_subsystem, dev);
 
 	mutex_lock(&nvme_subsystems_lock);
 	list_del(&subsys->entry);
@@ -1916,11 +1917,6 @@ static void nvme_free_subsystem(struct kref *ref)
 	__nvme_free_subsystem(subsys);
 }
 
-static void nvme_put_subsystem(struct nvme_subsystem *subsys)
-{
-	kref_put(&subsys->ref, nvme_free_subsystem);
-}
-
 static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
 {
 	struct nvme_subsystem *subsys;
@@ -1930,7 +1926,7 @@ static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
 	list_for_each_entry(subsys, &nvme_subsystems, entry) {
 		if (strcmp(subsys->subnqn, subsysnqn))
 			continue;
-		if (!kref_get_unless_zero(&subsys->ref))
+		if (!get_device(&subsys->dev))
 			continue;
 		return subsys;
 	}
@@ -1953,13 +1949,17 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 
 	INIT_LIST_HEAD(&subsys->ctrls);
 	INIT_LIST_HEAD(&subsys->nsheads);
-	kref_init(&subsys->ref);
 	nvme_init_subnqn(subsys, ctrl, id);
 	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
 	memcpy(subsys->model, id->mn, sizeof(subsys->model));
 	memcpy(subsys->firmware_rev, id->fr, sizeof(subsys->firmware_rev));
 	subsys->vendor_id = le16_to_cpu(id->vid);
 	mutex_init(&subsys->lock);
+	subsys->dev.class = nvme_subsys_class;
+	subsys->dev.parent = NULL;
+	subsys->dev.release = &nvme_free_subsystem;
+	dev_set_name(&subsys->dev, "subsys%u", subsys->instance);
+	device_initialize(&subsys->dev);
 
 	mutex_lock(&nvme_subsystems_lock);
 	found = __nvme_find_get_subsystem(subsys->subnqn);
@@ -1989,6 +1989,12 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
 	mutex_unlock(&subsys->lock);
 
+	if (!found) {
+		ret = device_add(&subsys->dev);
+		if (ret)
+			dev_err(ctrl->device,
+				"failed to register subsys device\n");
+	}
 	return 0;
 out_free_subsys:
 	kfree(subsys);
@@ -2891,7 +2897,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 			ns->disk->disk_name);
 
 	if (new) {
-		add_disk(ns->head->disk);
+		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
 		if (sysfs_create_group(&disk_to_dev(ns->head->disk)->kobj,
 				&nvme_ns_id_attr_group))
 			pr_warn("%s: failed to create sysfs group for identification\n",
@@ -3309,7 +3315,7 @@ static void nvme_free_ctrl(struct kref *kref)
 	ctrl->ops->free_ctrl(ctrl);
 
 	if (subsys)
-		nvme_put_subsystem(subsys);
+		put_device(&subsys->dev);
 }
 
 void nvme_put_ctrl(struct nvme_ctrl *ctrl)
@@ -3500,8 +3506,15 @@ int __init nvme_core_init(void)
 		goto unregister_chrdev;
 	}
 
+	nvme_subsys_class = class_create(THIS_MODULE, "nvme-subsys");
+	if (IS_ERR(nvme_subsys_class)) {
+		result = PTR_ERR(nvme_subsys_class);
+		goto destroy_class;
+	}
 	return 0;
 
+destroy_class:
+	class_destroy(nvme_class);
 unregister_chrdev:
 	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
 destroy_wq:
@@ -3512,6 +3525,7 @@ int __init nvme_core_init(void)
 void nvme_core_exit(void)
 {
 	ida_destroy(&nvme_subsystems_ida);
+	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
 	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
 	destroy_workqueue(nvme_wq);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f195ff..878ca87 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -202,11 +202,11 @@ struct nvme_ctrl {
 };
 
 struct nvme_subsystem {
+	struct device		dev;
 	struct list_head	entry;
 	struct mutex		lock;
 	struct list_head	ctrls;
 	struct list_head	nsheads;
-	struct kref		ref;
 	char			subnqn[NVMF_NQN_SIZE];
 	char			serial[20];
 	char			model[40];
-- 
1.8.5.6



From jthumshirn at suse.de  Mon Oct  2 07:35:03 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon,  2 Oct 2017 16:35:03 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <1506952559-1588-1-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
Message-ID: <20171002143503.14507-1-jthumshirn@suse.de>

When we do a path failover we should also reset the retry counter to 0
so we're not accidently failing I/O after a path failover.

Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1ef11ca0fc5e..f1d8b239b7b6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -118,6 +118,7 @@ static void nvme_failover_req(struct request *req)
 	blk_mq_end_request(req, 0);
 
 	nvme_reset_ctrl(ns->ctrl);
+	nvme_req(req)->retries = 0;
 	kblockd_schedule_work(&ns->head->requeue_work);
 }
 
-- 
2.12.3



From keith.busch at intel.com  Mon Oct  2 07:43:11 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 2 Oct 2017 08:43:11 -0600
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <20171002143503.14507-1-jthumshirn@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de>
Message-ID: <20171002144310.GR8463@localhost.localdomain>

On Mon, Oct 02, 2017 at 04:35:03PM +0200, Johannes Thumshirn wrote:
> When we do a path failover we should also reset the retry counter to 0
> so we're not accidently failing I/O after a path failover.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 1ef11ca0fc5e..f1d8b239b7b6 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -118,6 +118,7 @@ static void nvme_failover_req(struct request *req)
>  	blk_mq_end_request(req, 0);
>  
>  	nvme_reset_ctrl(ns->ctrl);
> +	nvme_req(req)->retries = 0;
>  	kblockd_schedule_work(&ns->head->requeue_work);
>  }

Won't this make it possible to retry indefinitely? We've removed the retry
time limit, so the only thing preventing inifinite retries on retryable
errors is the max retry limit.


From jthumshirn at suse.de  Mon Oct  2 07:45:34 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 02 Oct 2017 16:45:34 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <20171002143503.14507-1-jthumshirn@suse.de> (Johannes Thumshirn's
 message of "Mon, 2 Oct 2017 16:35:03 +0200")
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de>
Message-ID: <mqdtvzh38xd.fsf@linux-x5ow.site>

Gah, please ignore this one. I tested on the wrong device.

But FYI, when triggering a path failure in my RDMA setup (setting one
switch port down) I get these nice messages:
[ 1148.124063] nvme nvme0: SEND for CQE 0xffff8817c3720180 failed with
status transport retry counter exceeded (12)
[ 1148.180489] nvme nvme0: failed nvme_keep_alive_end_io error=10
[ 1148.187887] nvme nvme0: Reconnecting in 10 seconds...
[ 1148.194356] print_req_error: I/O error, dev nvme0n1, sector 1690128
[ 1148.194361] print_req_error: I/O error, dev nvme0n1, sector 1692168
[ 1148.194367] print_req_error: I/O error, dev nvme0n1, sector 1694208
[ 1148.379058] XFS (nvms0n1): writeback error on sector 1628688

the nvme ones are expected, but I don't really like to see writeback
errors from the FS here. Something's still a bit off.

Byte,
        Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Mon Oct  2 07:46:51 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 02 Oct 2017 16:46:51 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <20171002143503.14507-1-jthumshirn@suse.de> (Johannes Thumshirn's
 message of "Mon, 2 Oct 2017 16:35:03 +0200")
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de>
Message-ID: <mqdshf138v8.fsf@linux-x5ow.site>

Gah, please ignore this one. I tested on the wrong device.

But FYI, when triggering a path failure in my RDMA setup (setting one
switch port down) I get these nice messages:
[ 1148.124063] nvme nvme0: SEND for CQE 0xffff8817c3720180 failed with
status transport retry counter exceeded (12)
[ 1148.180489] nvme nvme0: failed nvme_keep_alive_end_io error=10
[ 1148.187887] nvme nvme0: Reconnecting in 10 seconds...
[ 1148.194356] print_req_error: I/O error, dev nvme0n1, sector 1690128
[ 1148.194361] print_req_error: I/O error, dev nvme0n1, sector 1692168
[ 1148.194367] print_req_error: I/O error, dev nvme0n1, sector 1694208
[ 1148.379058] XFS (nvms0n1): writeback error on sector 1628688

the nvme ones are expected, but I don't really like to see writeback
errors from the FS here. Something's still a bit off.

Byte,
        Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hare at suse.de  Mon Oct  2 08:08:21 2017
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 2 Oct 2017 17:08:21 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <20171002144310.GR8463@localhost.localdomain>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de>
 <20171002144310.GR8463@localhost.localdomain>
Message-ID: <d8080cdb-87fb-f008-a71c-2edf4e49117e@suse.de>

On 10/02/2017 04:43 PM, Keith Busch wrote:
> On Mon, Oct 02, 2017 at 04:35:03PM +0200, Johannes Thumshirn wrote:
>> When we do a path failover we should also reset the retry counter to 0
>> so we're not accidently failing I/O after a path failover.
>>
>> Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
>> ---
>>  drivers/nvme/host/core.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 1ef11ca0fc5e..f1d8b239b7b6 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -118,6 +118,7 @@ static void nvme_failover_req(struct request *req)
>>  	blk_mq_end_request(req, 0);
>>  
>>  	nvme_reset_ctrl(ns->ctrl);
>> +	nvme_req(req)->retries = 0;
>>  	kblockd_schedule_work(&ns->head->requeue_work);
>>  }
> 
> Won't this make it possible to retry indefinitely? We've removed the retry
> time limit, so the only thing preventing inifinite retries on retryable
> errors is the max retry limit.
> 
Please do check my earlier patch series.
The idea is to first retry paths, and then failover to other paths.
Once all paths are exhausted we'll requeue the I/O in nvme_make_request().
So from that we shouldn't have any infinite retries.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at lst.de  Mon Oct  2 09:15:32 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:15:32 +0200
Subject: [PATCH 1/6] nvme: display 'CMIC' controller attribute
In-Reply-To: <1506952559-1588-2-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-2-git-send-email-hare@suse.de>
Message-ID: <20171002161532.GA11048@lst.de>

On Mon, Oct 02, 2017 at 03:55:54PM +0200, Hannes Reinecke wrote:
> To figure out if a controller is multipath capable (ie if the
> controller supports namespace sharing) we need to expose the CMIC
> value to sysfs.

If you care about the multipath-capability of a controller you can
get it using nvme-cli.  Especially given that there is no good reason
to query it except for curiosity.


From hch at lst.de  Mon Oct  2 09:16:21 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:16:21 +0200
Subject: [PATCH 2/6] nvme: use 'nvmsXnZ' instead of 'nvm-subXnZ'
In-Reply-To: <1506952559-1588-3-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-3-git-send-email-hare@suse.de>
Message-ID: <20171002161621.GB11048@lst.de>

On Mon, Oct 02, 2017 at 03:55:55PM +0200, Hannes Reinecke wrote:
> Align with overall nvme device naming scheme.

It is very close to the normal /dev/nvme* devices, so I feat it will
create a lot of confusion.

But given that all device name suggestions so far suck I'd be happy
to take a stawpoll of linux-nvme subscribers :)


From hch at lst.de  Mon Oct  2 09:16:54 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:16:54 +0200
Subject: [PATCH 4/6] nvme: display 'NMIC' namespace attribute
In-Reply-To: <1506952559-1588-5-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-5-git-send-email-hare@suse.de>
Message-ID: <20171002161654.GC11048@lst.de>

On Mon, Oct 02, 2017 at 03:55:57PM +0200, Hannes Reinecke wrote:
> From: Johannes Thumshirn <jthumshirn at suse.de>
> 
> To figure out if a namespace is shared via multipath we need to expose
> the 'NMIC' value to sysfs.

What for?


From hch at lst.de  Mon Oct  2 09:18:08 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:18:08 +0200
Subject: [PATCH 5/6] nvme: Export subsystems to /sys/class/nvme-subsys
In-Reply-To: <1506952559-1588-6-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-6-git-send-email-hare@suse.de>
Message-ID: <20171002161808.GD11048@lst.de>

On Mon, Oct 02, 2017 at 03:55:58PM +0200, Hannes Reinecke wrote:
> Create a class for each subsystem to create a 'device' link
> for each subsystem device.

I thought about this as it seems useful.  The only downside is that
iff we run into subsystems with duplicate NQNs (or fake NQNs using
MN/SN) we are in deep trouble once this is exposed in sysfs.

Keith was the one most concerned about that, so I'd like him to
chime in.


From hch at lst.de  Mon Oct  2 09:19:29 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:19:29 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <20171002143503.14507-1-jthumshirn@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de>
Message-ID: <20171002161929.GE11048@lst.de>

On Mon, Oct 02, 2017 at 04:35:03PM +0200, Johannes Thumshirn wrote:
> When we do a path failover we should also reset the retry counter to 0
> so we're not accidently failing I/O after a path failover.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 1ef11ca0fc5e..f1d8b239b7b6 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -118,6 +118,7 @@ static void nvme_failover_req(struct request *req)
>  	blk_mq_end_request(req, 0);
>  
>  	nvme_reset_ctrl(ns->ctrl);
> +	nvme_req(req)->retries = 0;

Once we called blk_mq_end_request the request is gone, so this is
use after free.  And also very odd given that any multipath resubmit
will allocate a new request, so I can't see how it could help to start
with.


From keith.busch at intel.com  Mon Oct  2 09:20:39 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 2 Oct 2017 10:20:39 -0600
Subject: [PATCH 2/6] nvme: use 'nvmsXnZ' instead of 'nvm-subXnZ'
In-Reply-To: <20171002161621.GB11048@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-3-git-send-email-hare@suse.de>
 <20171002161621.GB11048@lst.de>
Message-ID: <20171002162038.GS8463@localhost.localdomain>

On Mon, Oct 02, 2017 at 06:16:21PM +0200, Christoph Hellwig wrote:
> On Mon, Oct 02, 2017 at 03:55:55PM +0200, Hannes Reinecke wrote:
> > Align with overall nvme device naming scheme.
> 
> It is very close to the normal /dev/nvme* devices, so I feat it will
> create a lot of confusion.
> 
> But given that all device name suggestions so far suck I'd be happy
> to take a stawpoll of linux-nvme subscribers :)

Of the options availble, +1 to Hannes' nvmsXnY.


From hch at lst.de  Mon Oct  2 09:22:54 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:22:54 +0200
Subject: [PATCH 6/6] nvme: ignore retries for multipath devices
In-Reply-To: <1506952559-1588-7-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-7-git-send-email-hare@suse.de>
Message-ID: <20171002162254.GA11497@lst.de>

> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 1ef11ca..ac7676a 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -201,7 +201,8 @@ static inline bool nvme_req_needs_retry(struct request *req)
>  		return false;
>  	if (nvme_req(req)->status & NVME_SC_DNR)
>  		return false;
> -	if (nvme_req(req)->retries >= nvme_max_retries)
> +	if (nvme_req(req)->retries >= nvme_max_retries &&
> +	    !(req->cmd_flags & REQ_NVME_MPATH))
>  		return false;
>  	return true;

All failover logic is inside a nvme_req_needs_retry() conditional,
so this change looks completely broken - it basically disables
failover.


From hch at lst.de  Mon Oct  2 09:25:26 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:25:26 +0200
Subject: [PATCH] nvme: reset retires after path failover
In-Reply-To: <mqdtvzh38xd.fsf@linux-x5ow.site>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <20171002143503.14507-1-jthumshirn@suse.de> <mqdtvzh38xd.fsf@linux-x5ow.site>
Message-ID: <20171002162526.GB11497@lst.de>

On Mon, Oct 02, 2017 at 04:45:34PM +0200, Johannes Thumshirn wrote:
> Gah, please ignore this one. I tested on the wrong device.
> 
> But FYI, when triggering a path failure in my RDMA setup (setting one
> switch port down) I get these nice messages:
> [ 1148.124063] nvme nvme0: SEND for CQE 0xffff8817c3720180 failed with
> status transport retry counter exceeded (12)
> [ 1148.180489] nvme nvme0: failed nvme_keep_alive_end_io error=10
> [ 1148.187887] nvme nvme0: Reconnecting in 10 seconds...
> [ 1148.194356] print_req_error: I/O error, dev nvme0n1, sector 1690128
> [ 1148.194361] print_req_error: I/O error, dev nvme0n1, sector 1692168
> [ 1148.194367] print_req_error: I/O error, dev nvme0n1, sector 1694208
> [ 1148.379058] XFS (nvms0n1): writeback error on sector 1628688
> 
> the nvme ones are expected, but I don't really like to see writeback
> errors from the FS here. Something's still a bit off.

I think the problem is our host-internal aborts that have the DNR
bit set.  My earier patches excluded DNR as a reason not to failover
and we'll either need to get back to that or remove DNR from these
sorts of errors.


From hch at lst.de  Mon Oct  2 09:42:18 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:42:18 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <1506952559-1588-4-git-send-email-hare@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
Message-ID: <20171002164218.GA11886@lst.de>

On Mon, Oct 02, 2017 at 03:55:56PM +0200, Hannes Reinecke wrote:
> When setting up a multipath device we need to claim the underlying
> block devices to avoid other systems and/or programs to access it.
> And we should be using the standard holders/slave sysfs relationship
> instead of the hand-crafted 'mpath' links.

This completely breaks backwards compatibility:

root at testvm:~# mkfs.xfs /dev/nvme0n1 -f
mkfs.xfs: cannot open /dev/nvme0n1: Device or resource busy

Also we really do not want and outstanding struct block_device reference
all the time - struct block_device should only have a reference if
the block device node is in use or a file system is mounted.  Avoiding
this case was the whole point of my refactor to store the gendisk
instead of the block_device in struct bio.


From keith.busch at intel.com  Mon Oct  2 09:53:18 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 2 Oct 2017 10:53:18 -0600
Subject: [PATCH 5/6] nvme: Export subsystems to /sys/class/nvme-subsys
In-Reply-To: <20171002161808.GD11048@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-6-git-send-email-hare@suse.de>
 <20171002161808.GD11048@lst.de>
Message-ID: <20171002165317.GT8463@localhost.localdomain>

On Mon, Oct 02, 2017 at 06:18:08PM +0200, Christoph Hellwig wrote:
> On Mon, Oct 02, 2017 at 03:55:58PM +0200, Hannes Reinecke wrote:
> > Create a class for each subsystem to create a 'device' link
> > for each subsystem device.
> 
> I thought about this as it seems useful.  The only downside is that
> iff we run into subsystems with duplicate NQNs (or fake NQNs using
> MN/SN) we are in deep trouble once this is exposed in sysfs.
> 
> Keith was the one most concerned about that, so I'd like him to
> chime in.

The driver will unbind from the controller if it detects an unsupported
duplicate nvme subsystem. The same logic will prevent duplciate sysfs
entries in this patch, so I think we're okay from that standpoint.

My concern was the user can't query or fix the controller that has the
duplicate subsystem name since we unbind from it. Unbinding is probably
the right thing to do, though, so the confused subsystem can be fixed
when the user isolates it from the others.

BTW, I just notice nvme_init_subsystem needs call nvme_put_subsystem in
the invalid duplicate subnqn case.


From hch at lst.de  Mon Oct  2 09:56:32 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 2 Oct 2017 18:56:32 +0200
Subject: [PATCH 5/6] nvme: Export subsystems to /sys/class/nvme-subsys
In-Reply-To: <20171002165317.GT8463@localhost.localdomain>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-6-git-send-email-hare@suse.de>
 <20171002161808.GD11048@lst.de> <20171002165317.GT8463@localhost.localdomain>
Message-ID: <20171002165632.GA12157@lst.de>

On Mon, Oct 02, 2017 at 10:53:18AM -0600, Keith Busch wrote:
> The driver will unbind from the controller if it detects an unsupported
> duplicate nvme subsystem. The same logic will prevent duplciate sysfs
> entries in this patch, so I think we're okay from that standpoint.
>
> My concern was the user can't query or fix the controller that has the
> duplicate subsystem name since we unbind from it. Unbinding is probably
> the right thing to do, though, so the confused subsystem can be fixed
> when the user isolates it from the others.
> 

Right now that's what we do.  But if this actually shows up in the wild
in e.g. consumer products we'll have to come up with a work around and
just allow that case and skip matches for it.  It would be a little ugly
but doable - once we expose the name in sysfs we are in deep trouble
with duplicate names.

> BTW, I just notice nvme_init_subsystem needs call nvme_put_subsystem in
> the invalid duplicate subnqn case.

We call __nvme_free_subsystem in that case, which should do the job.


From keith.busch at intel.com  Mon Oct  2 10:15:38 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 2 Oct 2017 11:15:38 -0600
Subject: [PATCH 5/6] nvme: Export subsystems to /sys/class/nvme-subsys
In-Reply-To: <20171002165632.GA12157@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-6-git-send-email-hare@suse.de>
 <20171002161808.GD11048@lst.de>
 <20171002165317.GT8463@localhost.localdomain>
 <20171002165632.GA12157@lst.de>
Message-ID: <20171002171538.GU8463@localhost.localdomain>

On Mon, Oct 02, 2017 at 06:56:32PM +0200, Christoph Hellwig wrote:
> 
> > BTW, I just notice nvme_init_subsystem needs call nvme_put_subsystem in
> > the invalid duplicate subnqn case.
> 
> We call __nvme_free_subsystem in that case, which should do the job.

We're calling __nvme_free_subsystem on the one that was allocated, not
the one that was found. The 'found' will have an additional reference
taken on it that isn't tracked.


From keith.busch at intel.com  Mon Oct  2 10:21:29 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 2 Oct 2017 11:21:29 -0600
Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <20171001074203.GA11115@lst.de>
References: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
 <20171001074203.GA11115@lst.de>
Message-ID: <20171002172128.GV8463@localhost.localdomain>

On Sun, Oct 01, 2017 at 09:42:03AM +0200, Christoph Hellwig wrote:
> This looks very convoluted, mostly because the existing code is
> doing weird things.  For one thing what is sq_dma_addr currently
> is not a DMA adddress - we either need the resource address
> for the ioremap, but we don't need to stash that away, and second
> the one programmed into the controller should be a pci_bus_addr_t.
> 
> Second we already have a nice PCI-layer helper called pci_bus_address
> to get the bus address for us and we should use it.
> 
> Something like the patch below should solve the issue:

Yah, calling this a DMA address was a misnomer and confusing.

> ---
> From b78f4164881125c4fecfdb87878d0120b2177c53 Mon Sep 17 00:00:00 2001
> From: Christoph Hellwig <hch at lst.de>
> Date: Sun, 1 Oct 2017 09:37:35 +0200
> Subject: nvme-pci: Use PCI bus address for data/queues in CMB
> 
> Currently, NVMe PCI host driver is programming CMB dma address as
> I/O SQs addresses. This results in failures on systems where 1:1
> outbound mapping is not used (example Broadcom iProc SOCs) because
> CMB BAR will be progammed with PCI bus address but NVMe PCI EP will
> try to access CMB using dma address.
> 
> To have CMB working on systems without 1:1 outbound mapping, we
> program PCI bus address for I/O SQs instead of dma address. This
> approach will work on systems with/without 1:1 outbound mapping.
> 
> Based on a report and previous patch from Abhishek Shah.
> 
> Fixes: 8ffaadf7 ("NVMe: Use CMB for the IO SQes if available")
> Cc: stable at vger.kernel.org
> Reported-by: Abhishek Shah <abhishek.shah at broadcom.com>
> Signed-off-by: Christoph Hellwig <hch at lst.de>

This looks good.

Reviewed-by: Keith Busch <keith.busch at intel.com>


From sagi at grimberg.me  Mon Oct  2 15:38:21 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 3 Oct 2017 01:38:21 +0300
Subject: [PATCH] nvmet_fc: Reduce work_q count
In-Reply-To: <20171001083932.GA878@infradead.org>
References: <20170927040958.5981-1-jsmart2021@gmail.com>
 <90b9711d-e279-443d-1693-dad24524cc74@grimberg.me>
 <e86009d5-2ca6-8de3-d142-781724bf00d8@gmail.com>
 <20171001083932.GA878@infradead.org>
Message-ID: <0fd8f917-5bfd-4d07-4c2e-6306e6a263f7@grimberg.me>


> And that's exactly what plain queue_work will give you.  If it doesn't
> do exactly what your open code logic does it has a bug that needs to
> be fixed.

I also think you are missing a put_cpu().


From sagi at grimberg.me  Mon Oct  2 15:45:10 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 3 Oct 2017 01:45:10 +0300
Subject: [PATCH 2/2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <991eab4b-d27c-2a17-ee14-0d9b11d9d2b1@broadcom.com>
References: <20170928153553.13993-1-jsmart2021@gmail.com>
 <20170928153553.13993-3-jsmart2021@gmail.com>
 <b5315d31-1d2c-b328-abd2-a734d4017372@grimberg.me>
 <991eab4b-d27c-2a17-ee14-0d9b11d9d2b1@broadcom.com>
Message-ID: <60908969-8934-e7c7-28b1-bc98fa62745a@grimberg.me>


>> fatal error handler was taking the assumption that that delete_ctrl
>> execution is asynchronous given that controller teardown is refcounted
>> by queues that are refcounted by inflight IO. This suggests that
>> controller actual free is async by nature, probably should have
>> documented it...
>>
>> Is fc's delete_ctrl blocks until all inflight IO is drained? I would
>> suggest to defer this blocking routine out of the fatal_error path like
>> rdma and loop. Is that something that breaks your design?
> 
> No - it really doesn't block waiting (like the host side) although it 
> may appear that way. Real difference is it processes the teardown in its 
> entirety and its possible, especially on light/idle load, the ref 
> counting could cause things to occur in the delete_ctrl context. Whereas 
> rdma and loop definitely convert over to another workq context for 
> teardown. Yes, I can do that too.? Yes, if there are requirements like 
> this for a transport - please add comments/documentation.? Although, as 
> you can see by this proposed patch, an implementation can be made in the 
> core that places no requirement on a transport.

I assume that we can change that to give some flexibility to the
transport implementation, so:

Acked-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Mon Oct  2 15:46:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 3 Oct 2017 01:46:42 +0300
Subject: [PATCH 1/2] block: genhd: add device_add_disk_with_groups
In-Reply-To: <20170928193637.24707-1-mwilck@suse.com>
References: <20170928193637.24707-1-mwilck@suse.com>
Message-ID: <18998c69-20f1-9abb-2297-0a9de015af95@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at gimberg.me>


From maxg at mellanox.com  Tue Oct  3 02:48:34 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Tue, 3 Oct 2017 12:48:34 +0300
Subject: Bug report: ref tag error while running IO > MDTS on nvme device
Message-ID: <25030a78-f59b-a9de-b570-e9e05c9f4411@mellanox.com>

Hi guys,
I would like to report a bug while running fio on local nvme device 
(SAMSUNG MZPLL1T6HEHP-00003) that was formated to use integrity:

nvme format /dev/nvme2n1 -n 1 -l 1 -i 2 -p 0 -m 0
nvme id-ns /dev/nvme2n1:

NVME Identify Namespace 1:
nsze    : 0xba4d4ab0
ncap    : 0xba4d4ab0
nuse    : 0xba4d4ab0
nsfeat  : 0
nlbaf   : 3
flbas   : 0x1
mc      : 0x3
dpc     : 0x1f
dps     : 0x2
nmic    : 0x1
rescap  : 0x7f
fpi     : 0x80
nawun   : 0
nawupf  : 0
nacwu   : 0
nabsn   : 0
nabo    : 0
nabspf  : 0
noiob   : 0
nvmcap  : 1600321314816
nguid   : 334842304a6000840025384100000002
eui64   : 002538e671000457
lbaf  0 : ms:0   lbads:9  rp:0x1
lbaf  1 : ms:8   lbads:9  rp:0x3 (in use)
lbaf  2 : ms:0   lbads:12 rp:0
lbaf  3 : ms:8   lbads:12 rp:0x2

running traffic up to IO size <= 128KB works fine but running bigger IO 
that the MDTS caused many ref tag errors (for example):

[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 324043008 
(rcvd 0)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2929058096 
(rcvd 1241579520)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2292844816 
(rcvd 2483027968)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2523145472 
(rcvd 243859455)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1177656832 
(rcvd 15400959)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 187312640 
(rcvd 243859455)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2344798720 
(rcvd 126418943)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2892037376 
(rcvd 126418943)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2292845056 
(rcvd 143196159)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1237630256 
(rcvd 1241579520)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1237630208 
(rcvd 143196159)
[Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2986034432 
(rcvd 243859455)
[Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 881629440 
(rcvd 15400959)
[Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 1889589504 
(rcvd 243859455)
[Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 1889589504 
(rcvd 243859455)

haven't debugged it yet, but if anyone has an idea for the root cause, 
it would be great.

Thanks,
-Max.



From hare at suse.de  Tue Oct  3 03:00:16 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 3 Oct 2017 12:00:16 +0200
Subject: [PATCH 4/6] nvme: display 'NMIC' namespace attribute
In-Reply-To: <20171002161654.GC11048@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-5-git-send-email-hare@suse.de>
 <20171002161654.GC11048@lst.de>
Message-ID: <e5f46d2d-9473-c99c-9319-8567567b6994@suse.de>

On 10/02/2017 06:16 PM, Christoph Hellwig wrote:
> On Mon, Oct 02, 2017 at 03:55:57PM +0200, Hannes Reinecke wrote:
>> From: Johannes Thumshirn <jthumshirn at suse.de>
>>
>> To figure out if a namespace is shared via multipath we need to expose
>> the 'NMIC' value to sysfs.
> 
> What for?
> 
Do stop systemd/udev doing weird things on the device.
If we have the 'NMIC' attribute in sysfs we can evaluate it during event
handling, and set the 'SYSTEMD_READY=0' flag if NMIC=1.
By setting this attribute udev is instructed to not do any fancy
checking, and systemd will leave it alone.
Otherwise systemd will happily forward the block device to all attached
services, btrfs will start checking the device, blkid will be scanning
for filesystems, swap will declaring this device as a valid swap space
and the like.

One of the painful lessons learned when moving multipath-tools to
systemd; we absolutely need an indicator in sysfs to handle multipath
devices race-free.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct  3 03:02:38 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 3 Oct 2017 12:02:38 +0200
Subject: [PATCH 6/6] nvme: ignore retries for multipath devices
In-Reply-To: <20171002162254.GA11497@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-7-git-send-email-hare@suse.de>
 <20171002162254.GA11497@lst.de>
Message-ID: <9d603720-2f86-ab3c-6ba3-9d57afb3568e@suse.de>

On 10/02/2017 06:22 PM, Christoph Hellwig wrote:
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 1ef11ca..ac7676a 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -201,7 +201,8 @@ static inline bool nvme_req_needs_retry(struct request *req)
>>  		return false;
>>  	if (nvme_req(req)->status & NVME_SC_DNR)
>>  		return false;
>> -	if (nvme_req(req)->retries >= nvme_max_retries)
>> +	if (nvme_req(req)->retries >= nvme_max_retries &&
>> +	    !(req->cmd_flags & REQ_NVME_MPATH))
>>  		return false;
>>  	return true;
> 
> All failover logic is inside a nvme_req_needs_retry() conditional,
> so this change looks completely broken - it basically disables
> failover.
> 
Not in our tests.
Without this patch we'd been seeing I/O errors during failover; with
this patch I/O continues on the failover path.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct  3 03:08:26 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 3 Oct 2017 12:08:26 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171002164218.GA11886@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
Message-ID: <ece83015-0def-8e36-5645-7be00be95c40@suse.de>

On 10/02/2017 06:42 PM, Christoph Hellwig wrote:
> On Mon, Oct 02, 2017 at 03:55:56PM +0200, Hannes Reinecke wrote:
>> When setting up a multipath device we need to claim the underlying
>> block devices to avoid other systems and/or programs to access it.
>> And we should be using the standard holders/slave sysfs relationship
>> instead of the hand-crafted 'mpath' links.
> 
> This completely breaks backwards compatibility:
> 
> root at testvm:~# mkfs.xfs /dev/nvme0n1 -f
> mkfs.xfs: cannot open /dev/nvme0n1: Device or resource busy
> 
> Also we really do not want and outstanding struct block_device reference
> all the time - struct block_device should only have a reference if
> the block device node is in use or a file system is mounted.  Avoiding
> this case was the whole point of my refactor to store the gendisk
> instead of the block_device in struct bio.
> 
Uh-oh.

And now the fun begins.

With this patch dm-multipath will ignore this device, so it's
_impossible_ to setup any device-mapper or md device on top of the
'nvme' device.
Without this patch md will happily autostart any RAID device on these
devices when it find a signature.
Or the user can mount it.
Or swap can attach it as a swapspace.

_AND_ you can do the same with the subsystem device, too, without any
indication that anything is amiss.

Do we really want this?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at lst.de  Tue Oct  3 04:49:24 2017
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 3 Oct 2017 13:49:24 +0200
Subject: [PATCH 4/6] nvme: display 'NMIC' namespace attribute
In-Reply-To: <e5f46d2d-9473-c99c-9319-8567567b6994@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-5-git-send-email-hare@suse.de>
 <20171002161654.GC11048@lst.de>
 <e5f46d2d-9473-c99c-9319-8567567b6994@suse.de>
Message-ID: <20171003114924.GA24650@lst.de>

On Tue, Oct 03, 2017 at 12:00:16PM +0200, Hannes Reinecke wrote:
> Do stop systemd/udev doing weird things on the device.

Please define weird things, preferably actual practical wierd things
you've observed with this implementation and then document them here
in the patch description.

> If we have the 'NMIC' attribute in sysfs we can evaluate it during event
> handling, and set the 'SYSTEMD_READY=0' flag if NMIC=1.

Bonus points of including such sniplets in the patch description,
similar to how I offered udev rules in the main multipath path.

> One of the painful lessons learned when moving multipath-tools to
> systemd; we absolutely need an indicator in sysfs to handle multipath
> devices race-free.

dm-multipath devices require userspace setup, nvme-mpath ones don't.
That's a huge difference that would make udev rules a lot simpler.
But if they don't I'd like to see a good explanation here, both to
understand why you'd want this flag, and also if there is a way to
just do the right thing from the kernel.


From hch at lst.de  Tue Oct  3 04:53:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 3 Oct 2017 13:53:55 +0200
Subject: [PATCH 6/6] nvme: ignore retries for multipath devices
In-Reply-To: <9d603720-2f86-ab3c-6ba3-9d57afb3568e@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-7-git-send-email-hare@suse.de>
 <20171002162254.GA11497@lst.de>
 <9d603720-2f86-ab3c-6ba3-9d57afb3568e@suse.de>
Message-ID: <20171003115355.GB24650@lst.de>

On Tue, Oct 03, 2017 at 12:02:38PM +0200, Hannes Reinecke wrote:
> >>  	if (nvme_req(req)->status & NVME_SC_DNR)
> >>  		return false;
> >> -	if (nvme_req(req)->retries >= nvme_max_retries)
> >> +	if (nvme_req(req)->retries >= nvme_max_retries &&
> >> +	    !(req->cmd_flags & REQ_NVME_MPATH))
> >>  		return false;
> >>  	return true;
> > 
> > All failover logic is inside a nvme_req_needs_retry() conditional,
> > so this change looks completely broken - it basically disables
> > failover.
> > 
> Not in our tests.
> Without this patch we'd been seeing I/O errors during failover; with
> this patch I/O continues on the failover path.

http://git.infradead.org/users/hch/block.git/blob/refs/heads/nvme-mpath:/drivers/nvme/host/core.c#l208

210	if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
211		if (nvme_req_needs_failover(req)) {
212			nvme_failover_req(req);
213			return;
214		 }

The only call to nvme_failover_req is guarded by nvme_req_needs_retry,
and you change needs_retry to return true for MPATH requests that
exceed the number of retries.  I just don't see how we'd hit the
max_retries count, as each retry before should have already taken
nvme_req_needs_failover before.  What error code do you see this
with?  What kinds of device/setup?


From hch at lst.de  Tue Oct  3 04:55:07 2017
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 3 Oct 2017 13:55:07 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
Message-ID: <20171003115506.GC24650@lst.de>

On Tue, Oct 03, 2017 at 12:08:26PM +0200, Hannes Reinecke wrote:
> With this patch dm-multipath will ignore this device, so it's
> _impossible_ to setup any device-mapper or md device on top of the
> 'nvme' device.
> Without this patch md will happily autostart any RAID device on these
> devices when it find a signature.
> Or the user can mount it.
> Or swap can attach it as a swapspace.
> 
> _AND_ you can do the same with the subsystem device, too, without any
> indication that anything is amiss.
> 
> Do we really want this?

I though we agreed last week to do dynamic claiming.

That is if someone claims the multipath-device we extend that claim to
all /dev/nvmeXnY devices, and if someone claims one of the /dev/nvmeXnY
devices we extent it to the multipath node.  That should solve all of
the above issues.


From keith.busch at intel.com  Tue Oct  3 07:25:15 2017
From: keith.busch at intel.com (Keith Busch)
Date: Tue, 3 Oct 2017 08:25:15 -0600
Subject: Bug report: ref tag error while running IO > MDTS on nvme device
In-Reply-To: <25030a78-f59b-a9de-b570-e9e05c9f4411@mellanox.com>
References: <25030a78-f59b-a9de-b570-e9e05c9f4411@mellanox.com>
Message-ID: <20171003142515.GA16678@localhost.localdomain>

Hi Max,

Is this behavior new, or was this always broken?

Since you mention MDTS, I wonder if this device is counting the separate
metadata as part of the transfer and doing the wrong thing as a result?
We're only supposed to count the metadata size in consideration of MDTS
if metadata is interleaved.

On Tue, Oct 03, 2017 at 12:48:34PM +0300, Max Gurtovoy wrote:
> Hi guys,
> I would like to report a bug while running fio on local nvme device (SAMSUNG
> MZPLL1T6HEHP-00003) that was formated to use integrity:
> 
> nvme format /dev/nvme2n1 -n 1 -l 1 -i 2 -p 0 -m 0
> nvme id-ns /dev/nvme2n1:
> 
> NVME Identify Namespace 1:
> nsze    : 0xba4d4ab0
> ncap    : 0xba4d4ab0
> nuse    : 0xba4d4ab0
> nsfeat  : 0
> nlbaf   : 3
> flbas   : 0x1
> mc      : 0x3
> dpc     : 0x1f
> dps     : 0x2
> nmic    : 0x1
> rescap  : 0x7f
> fpi     : 0x80
> nawun   : 0
> nawupf  : 0
> nacwu   : 0
> nabsn   : 0
> nabo    : 0
> nabspf  : 0
> noiob   : 0
> nvmcap  : 1600321314816
> nguid   : 334842304a6000840025384100000002
> eui64   : 002538e671000457
> lbaf  0 : ms:0   lbads:9  rp:0x1
> lbaf  1 : ms:8   lbads:9  rp:0x3 (in use)
> lbaf  2 : ms:0   lbads:12 rp:0
> lbaf  3 : ms:8   lbads:12 rp:0x2
> 
> running traffic up to IO size <= 128KB works fine but running bigger IO that
> the MDTS caused many ref tag errors (for example):
> 
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 324043008
> (rcvd 0)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2929058096
> (rcvd 1241579520)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2292844816
> (rcvd 2483027968)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2523145472
> (rcvd 243859455)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1177656832
> (rcvd 15400959)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 187312640
> (rcvd 243859455)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2344798720
> (rcvd 126418943)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2892037376
> (rcvd 126418943)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2292845056
> (rcvd 143196159)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1237630256
> (rcvd 1241579520)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 1237630208
> (rcvd 143196159)
> [Tue Oct  3 12:34:25 2017] nvme2n1: ref tag error at location 2986034432
> (rcvd 243859455)
> [Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 881629440
> (rcvd 15400959)
> [Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 1889589504
> (rcvd 243859455)
> [Tue Oct  3 12:34:45 2017] nvme2n1: ref tag error at location 1889589504
> (rcvd 243859455)
> 
> haven't debugged it yet, but if anyone has an idea for the root cause, it
> would be great.


From keith.busch at intel.com  Tue Oct  3 08:06:49 2017
From: keith.busch at intel.com (Keith Busch)
Date: Tue, 3 Oct 2017 09:06:49 -0600
Subject: [PATCH] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <CAKUFe6YHJ+jBgxtEy+YTTDQtphtOEz6+xfBXdowsFKy7ZMqW3w@mail.gmail.com>
References: <1506662966-10865-1-git-send-email-abhishek.shah@broadcom.com>
 <20170929144242.GN8463@localhost.localdomain>
 <CAKUFe6YHJ+jBgxtEy+YTTDQtphtOEz6+xfBXdowsFKy7ZMqW3w@mail.gmail.com>
Message-ID: <20171003150649.GA2195@localhost.localdomain>

On Sat, Sep 30, 2017 at 02:30:16PM +0530, Abhishek Shah wrote:
> > On a similar note, we also break CMB usage in virutalization with direct
> > assigned devices: the guest doesn't know the host physical bus address,
> > so it sets the CMB queue address incorrectly there, too. I don't know of
> > a way to fix that other than disabling CMB.
> 
> I don't have much idea on CMB usage in virtualization... will let
> someone else comment on this.

It's okay, I think it's a gap in the NVMe specification that would allow
such usage.


From maxg at mellanox.com  Tue Oct  3 08:19:34 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Tue, 3 Oct 2017 18:19:34 +0300
Subject: Bug report: ref tag error while running IO > MDTS on nvme device
In-Reply-To: <20171003142515.GA16678@localhost.localdomain>
References: <25030a78-f59b-a9de-b570-e9e05c9f4411@mellanox.com>
 <20171003142515.GA16678@localhost.localdomain>
Message-ID: <c246b05d-767d-a2a8-bdb3-6f7f5a7bf322@mellanox.com>



On 10/3/2017 5:25 PM, Keith Busch wrote:
> Hi Max,

Hi Keith,

> 
> Is this behavior new, or was this always broken?

Good question.
Haven't tried older kernels (only 4.13.0).

> 
> Since you mention MDTS, I wonder if this device is counting the separate
> metadata as part of the transfer and doing the wrong thing as a result?
> We're only supposed to count the metadata size in consideration of MDTS
> if metadata is interleaved.

mdts is 5 (max_hw_sectors_kb is 128) so I guess it's ok from device 
point of view.


From hare at suse.de  Tue Oct  3 09:01:10 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 3 Oct 2017 18:01:10 +0200
Subject: [PATCH 4/6] nvme: display 'NMIC' namespace attribute
In-Reply-To: <20171003114924.GA24650@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-5-git-send-email-hare@suse.de>
 <20171002161654.GC11048@lst.de>
 <e5f46d2d-9473-c99c-9319-8567567b6994@suse.de>
 <20171003114924.GA24650@lst.de>
Message-ID: <a99239fb-2f35-a53a-2a7b-52b324e76d86@suse.de>

On 10/03/2017 01:49 PM, Christoph Hellwig wrote:
> On Tue, Oct 03, 2017 at 12:00:16PM +0200, Hannes Reinecke wrote:
>> Do stop systemd/udev doing weird things on the device.
> 
> Please define weird things, preferably actual practical wierd things
> you've observed with this implementation and then document them here
> in the patch description.
> 
Any event will be passed to systemd, and ends up being passed as a
'target' to the systemd service evaluation routines.
One of the services happens to be the systemd-fsck at .service, which will
be calling fsck on every device found in /etc/fstab.

If now this device is specified with an fs uuid (as it's common
nowadays) systemd will be invoking fsck on each device carrying this UUID.
As we're having _two_ devices with the same UUID (at the very least;
actually it's 'number of paths + 1'), systemd will be invoking fsck on
_each_ of those devices.
All pointing to the same underlying namespace.
I'd be very curious to figure out how fsck deals with such a situation.

And then systemd will try to _mount_ each of those devices, which is
where the real fun begins. Depending on the timing it might chose to
mount any of those devices, more often than not at the same time as
another fsck instance is running on another path.
Again, I sincerely doubt that either mount nor fsck is able to handle
these situations.

The errors from these scenarios will vary wildly, from simple I/O errors
to filesystems corruption.

Hence it's adviseable to set the 'SYSTEMD_READY=0' flag on all
underlying paths, and only forward the 'real' multipathed device to
systemd for evaluation.
But this needs to be done _before_ any other event has a chance to
interrupt here.
_And_ we cannot call an external userspace program here, as this is
multipathing, and we might be called for an 'add' event when all paths
are down, at which point we cannot access the root-fs and udev will
stall trying to load the said program.
Hence the event will never finished processing, the path will never
recover, and your system is dead with all paths running.
Not good.

>> If we have the 'NMIC' attribute in sysfs we can evaluate it during event
>> handling, and set the 'SYSTEMD_READY=0' flag if NMIC=1.
> 
> Bonus points of including such sniplets in the patch description,
> similar to how I offered udev rules in the main multipath path.
> 
Okay, will be doing so.

>> One of the painful lessons learned when moving multipath-tools to
>> systemd; we absolutely need an indicator in sysfs to handle multipath
>> devices race-free.
> 
> dm-multipath devices require userspace setup, nvme-mpath ones don't.
> That's a huge difference that would make udev rules a lot simpler.
> But if they don't I'd like to see a good explanation here, both to
> understand why you'd want this flag, and also if there is a way to
> just do the right thing from the kernel.
> 

As mentioned above, yes, the udev rules will be massively simpler.
Essentially, the rule just boils down to setting the 'SYSTEMD_READY=0'
flag when NMIC is set.
But for that we need to figure out _if_ that bit is set, and we need to
do so without any external userspace program.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From ckulkarnilinux at gmail.com  Tue Oct  3 12:24:24 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue,  3 Oct 2017 12:24:24 -0700
Subject: [PATCH V4] nvme-pci: add SGL support 
Message-ID: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>

This is the fourth version of the patch for adding the SGL support
for nvme-pci driver.

Changes since v3:-

1. Modify SGL helper functions and add prefix nvme_pci_sgl.
2. Use average request physical segment(s) size to conditionally use SGLs.
   This approach is different from the original one, the original approach
   was based no the IO size. The new approach considers IO size and 
   number of physical segments present in the request to conditionally
   use SGLs.
3. Adjust the patch for Linux 4.14-rc3.

Changes since v2:-

1. Adjust the patch for latest nvme driver changes.
2. For initial submission use the default virt boundary values.
    
In future, work on the patch series for relaxing the virt boundary
for each transport.

[PATCH V3] nvme-pci: add sgl support

Changes since the v1:-

1. Introduced nvme controller virt_boundary mask. For controllers that
   can handle arbitrarily sized bios (e.g advanced RDMA and PCI ctrls)
   we can allow the block layer to pass us gaps by setting the 
   queue.virt_boundary_mask according to each ctrl constraints.
2. Simplified code for setting up the SGLs.
3. Added SGL helper functions.
4. Modified commit log.
5. Use the newly introduced virt_boundary mask and clear its 
   value for SGL capable controllers in PCIe driver.

For reference v1 details:-

This adds SGL support for NVMe PCIe driver which is
reimplementation of the initial version provided by
Rajiv Shanmugam Madeswaran(smrajiv15 at gmail.com):-
"block: nvme-core: Scatter gather list support in the 
NVMe Block Driver".

In order to enable SGL mode for the driver, the user can set the 
sgl_threshold module parameter. This parameter can be used
in the two modes:-

1. Allow all IOs to use SGL mode. (set sgl_threshold to 1). 
2. Set up the IO size threshold to determine whether to use 
    SGLs or PRPs for each IO. (set sgl_threshold to 4096 to
    use SGL only for IOs which are >= 4096 in the size).

[PATCH V4 1/2] nvme: add SGL controller capability constant
[PATCH V4 2/2] nvme-pci: add SGL support

-Regards,
Chaitanya


From ckulkarnilinux at gmail.com  Tue Oct  3 12:24:25 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue,  3 Oct 2017 12:24:25 -0700
Subject: [PATCH V4 1/2] nvme: add SGL controller capability constant
In-Reply-To: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1507058666-15298-2-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a constant definition which can be used to determine whether
NVMe Controller supports SGL mode for IOs.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 include/linux/nvme.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9310ce77..22bbcfb 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -466,6 +466,8 @@ enum nvme_opcode {
 	nvme_cmd_resv_release	= 0x15,
 };
 
+#define NVME_CTRL_SUPP_SGL     0x1
+
 /*
  * Descriptor subtype - lower 4 bits of nvme_(keyed_)sgl_desc identifier
  *
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct  3 12:24:26 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue,  3 Oct 2017 12:24:26 -0700
Subject: [PATCH V4 2/2] nvme-pci: add SGL support
In-Reply-To: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1507058666-15298-3-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds SGL support for NVMe PCIe driver, based on an earlier patch
from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
refactors the original code and adds new module parameter sgl_threshold
to determine whether to use SGL or PRP for IOs.

The usage of SGLs is controlled by the sgl_threshold module parameter,
which allows to conditionally use SGLs if average request segment size
(avg_seg_size) is greater than sgl_threshold. In the original patch
the decision of using SGLs was dependent only on the IO size, with new
the approach we consider not only IO size but also the number of
physical segments present in the IO.

We calculate avg_seg_size based on request payload bytes and number
of physical segments present in the request.

For e.g.:-

1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.

3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 223 ++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 198 insertions(+), 25 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cb73bc8..1876fdd 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -45,6 +45,8 @@
  */
 #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
 
+#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+
 static int use_threaded_interrupts;
 module_param(use_threaded_interrupts, int, 0);
 
@@ -57,6 +59,10 @@
 MODULE_PARM_DESC(max_host_mem_size_mb,
 	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
 
+static unsigned int sgl_threshold = SZ_32K;
+module_param(sgl_threshold, uint, 0644);
+MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this size");
+
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
 static const struct kernel_param_ops io_queue_depth_ops = {
 	.set = io_queue_depth_set,
@@ -178,6 +184,7 @@ struct nvme_queue {
 struct nvme_iod {
 	struct nvme_request req;
 	struct nvme_queue *nvmeq;
+	bool use_sgl;
 	int aborted;
 	int npages;		/* In the PRP list. 0 means small pool in use */
 	int nents;		/* Used in scatterlist */
@@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
 	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
 }
 
+/*
+ * Calculates the number of pages needed for the SGL segments. For example a 4k
+ * page can accommodate 256 SGL descriptors.
+ */
+static int nvme_npages_sgl(unsigned int num_seg)
+{
+	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
+}
+
 static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
-		unsigned int size, unsigned int nseg)
+		unsigned int size, unsigned int nseg, bool use_sgl)
 {
-	return sizeof(__le64 *) * nvme_npages(size, dev) +
-			sizeof(struct scatterlist) * nseg;
+	size_t alloc_size;
+
+	if (use_sgl)
+		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
+	else
+		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
+
+	return alloc_size + sizeof(struct scatterlist) * nseg;
 }
 
-static unsigned int nvme_cmd_size(struct nvme_dev *dev)
+static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
 {
 	return sizeof(struct nvme_iod) +
-		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
+		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
 }
 
 static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
@@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
 	nvmeq->sq_tail = tail;
 }
 
-static __le64 **iod_list(struct request *req)
+static void **iod_list(struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
+	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
 }
 
 static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
@@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 	unsigned int size = blk_rq_payload_bytes(rq);
 
 	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
-		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
+		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
+				iod->use_sgl);
+
+		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
 		if (!iod->sg)
 			return BLK_STS_RESOURCE;
 	} else {
@@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	const int last_prp = dev->ctrl.page_size / 8 - 1;
+	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
+	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
 	int i;
-	__le64 **list = iod_list(req);
-	dma_addr_t prp_dma = iod->first_dma;
 
 	if (iod->npages == 0)
-		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
+		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
+
 	for (i = 0; i < iod->npages; i++) {
-		__le64 *prp_list = list[i];
-		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
-		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
-		prp_dma = next_prp_dma;
+		void *addr = iod_list(req)[i];
+
+		if (iod->use_sgl) {
+			struct nvme_sgl_desc *sg_list = addr;
+
+			next_dma_addr =
+				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
+		} else {
+			__le64 *prp_list = addr;
+
+			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
+		}
+
+		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
+		dma_addr = next_dma_addr;
 	}
 
 	if (iod->sg != iod->inline_sg)
@@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
 	}
 }
 
-static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
+static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmnd)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
 	struct dma_pool *pool;
@@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	u32 page_size = dev->ctrl.page_size;
 	int offset = dma_addr & (page_size - 1);
 	__le64 *prp_list;
-	__le64 **list = iod_list(req);
+	void **list = iod_list(req);
 	dma_addr_t prp_dma;
 	int nprps, i;
 
+	iod->use_sgl = false;
+
 	length -= (page_size - offset);
 	if (length <= 0) {
 		iod->first_dma = 0;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	dma_len -= (page_size - offset);
@@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 
 	if (length <= page_size) {
 		iod->first_dma = dma_addr;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	nprps = DIV_ROUND_UP(length, page_size);
@@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 		dma_len = sg_dma_len(sg);
 	}
 
+done:
+	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
+	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
+
 	return BLK_STS_OK;
 
  bad_sgl:
@@ -643,6 +686,130 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	return BLK_STS_IOERR;
 }
 
+static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
+		struct scatterlist *sg)
+{
+	sge->addr = cpu_to_le64(sg_dma_address(sg));
+	sge->length = cpu_to_le32(sg_dma_len(sg));
+	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr, int entries)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
+	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(PAGE_SIZE);
+	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
+}
+
+static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmd)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	int length = blk_rq_payload_bytes(req);
+	struct dma_pool *pool;
+	struct nvme_sgl_desc *sg_list;
+	struct scatterlist *sg = iod->sg;
+	int entries = iod->nents, i = 0;
+	dma_addr_t sgl_dma;
+
+	iod->use_sgl = true;
+
+	cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as SGL */
+
+	if (length == sg_dma_len(sg)) {
+		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
+		return BLK_STS_OK;
+	}
+
+	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
+		pool = dev->prp_small_pool;
+		iod->npages = 0;
+	} else {
+		pool = dev->prp_page_pool;
+		iod->npages = 1;
+	}
+
+	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+	if (!sg_list) {
+		iod->npages = -1;
+		return BLK_STS_RESOURCE;
+	}
+
+	iod_list(req)[0] = sg_list;
+	iod->first_dma = sgl_dma;
+
+	if (entries <= SGES_PER_PAGE) {
+		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
+
+		for (i = 0; i < entries; i++) {
+			nvme_pci_sgl_set_data(&sg_list[i], sg);
+			length -= sg_dma_len(sg);
+			sg = sg_next(sg);
+		}
+
+		WARN_ON(length > 0);
+		return BLK_STS_OK;
+	}
+
+	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
+
+	do {
+		if (i == SGES_PER_PAGE) {
+			struct nvme_sgl_desc *old_sg_desc = sg_list;
+			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
+
+			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+			if (!sg_list)
+				return BLK_STS_RESOURCE;
+
+			i = 0;
+			iod_list(req)[iod->npages++] = sg_list;
+			sg_list[i++] = *link;
+
+			if (entries < SGES_PER_PAGE)
+				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
+			else
+				nvme_pci_sgl_set_seg(link, sgl_dma);
+		}
+
+		nvme_pci_sgl_set_data(&sg_list[i], sg);
+
+		length -= sg_dma_len(sg);
+		sg = sg_next(sg);
+		entries--;
+	} while (length > 0);
+
+	WARN_ON(entries > 0);
+	return BLK_STS_OK;
+}
+
+static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	unsigned int avg_seg_size;
+
+	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
+			blk_rq_nr_phys_segments(req));
+
+	if (!(dev->ctrl.sgls & NVME_CTRL_SUPP_SGL))
+		return false;
+	if (!iod->nvmeq->qid)
+		return false;
+	if (!sgl_threshold || avg_seg_size < sgl_threshold)
+		return false;
+	return true;
+}
+
+
 static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 		struct nvme_command *cmnd)
 {
@@ -662,7 +829,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 				DMA_ATTR_NO_WARN))
 		goto out;
 
-	ret = nvme_setup_prps(dev, req);
+	if (nvme_pci_use_sgls(dev, req))
+		ret = nvme_setup_sgls(dev, req, &cmnd->rw);
+	else
+		ret = nvme_setup_prps(dev, req, &cmnd->rw);
+
 	if (ret != BLK_STS_OK)
 		goto out_unmap;
 
@@ -682,8 +853,6 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 			goto out_unmap;
 	}
 
-	cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
-	cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
 	if (blk_integrity_rq(req))
 		cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
 	return BLK_STS_OK;
@@ -1379,7 +1548,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
-		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
+		dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
 		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
 		dev->admin_tagset.driver_data = dev;
 
@@ -1906,7 +2075,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
 		dev->tagset.numa_node = dev_to_node(dev->dev);
 		dev->tagset.queue_depth =
 				min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
-		dev->tagset.cmd_size = nvme_cmd_size(dev);
+		dev->tagset.cmd_size = nvme_cmd_size(dev, false);
+		if ((dev->ctrl.sgls & NVME_CTRL_SUPP_SGL) && sgl_threshold) {
+			dev->tagset.cmd_size = max(dev->tagset.cmd_size,
+					nvme_cmd_size(dev, true));
+		}
 		dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
 		dev->tagset.driver_data = dev;
 
-- 
1.8.3.1



From jsmart2021 at gmail.com  Tue Oct  3 12:40:40 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 3 Oct 2017 12:40:40 -0700
Subject: [PATCH] nvmet_fc: Reduce work_q count
In-Reply-To: <20171001083932.GA878@infradead.org>
References: <20170927040958.5981-1-jsmart2021@gmail.com>
 <90b9711d-e279-443d-1693-dad24524cc74@grimberg.me>
 <e86009d5-2ca6-8de3-d142-781724bf00d8@gmail.com>
 <20171001083932.GA878@infradead.org>
Message-ID: <147c11f9-24cc-7328-80e9-409cbb0e9e46@gmail.com>

On 10/1/2017 1:39 AM, Christoph Hellwig wrote:
> On Sat, Sep 30, 2017 at 11:32:39AM -0700, James Smart wrote:
>> Because we're still trying to get cpu affinity on the connection resources
>> that will be shared by the io on the same queue.
> 
> And that's exactly what plain queue_work will give you.  If it doesn't
> do exactly what your open code logic does it has a bug that needs to
> be fixed.
> 
I don't understand how queuing work alone, which can go to any cpu 
(current preferred) helps with avoiding cross-cpu access to the same 
transport/lldd hw queue data structures.

But, as stated. A moot point now. I'll remove this whole IN_ISR, thus 
work_q item, shortly.

-- james



From jsmart2021 at gmail.com  Tue Oct  3 13:01:39 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 3 Oct 2017 13:01:39 -0700
Subject: [PATCH] nvme_fc: correct io timeout behavior
In-Reply-To: <20171001084700.GB878@infradead.org>
References: <20170927041317.6575-1-jsmart2021@gmail.com>
 <20171001084700.GB878@infradead.org>
Message-ID: <ddf42345-7e87-f618-f93d-ab756648305d@gmail.com>

On 10/1/2017 1:47 AM, Christoph Hellwig wrote:
>> --- a/drivers/nvme/host/fc.c
>> +++ b/drivers/nvme/host/fc.c
>> @@ -1374,7 +1374,7 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>>   				sizeof(op->rsp_iu), DMA_FROM_DEVICE);
>>   
>>   	if (atomic_read(&op->state) == FCPOP_STATE_ABORTED)
>> -		status = cpu_to_le16((NVME_SC_ABORT_REQ | NVME_SC_DNR) << 1);
>> +		status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);
> 
> So the DNR bit goes away, ok.

yes.  The aborted state is happening on io aborted due to resets, DNR 
was causing it to fail back to the app rather than be rescheduled for 
post reconnect.


>> @@ -1445,15 +1445,22 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
>>   		goto check_error;
>>   	}
>>   
>> +	if (unlikely(!status && op->flags & FCOP_FLAGS_TERMIO))
>> +		status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);
> 
> We grow another aborted case.  

actually, I believe this is overlapped on the above FCPOP_STATE_ABORTED 
flag checked above. So I believe I can remove this.

Note: these FCPOP_STATE_ABORTED/FCOP_FLAGS_TERMIO should only be set in 
cases where the controller is being torn down after a transport error, 
io timeout, or loss of connectivity.


> And btw, we really should not use
> NVME_SC_ABORT_REQ here because the command never reched the controller
> to actually get a nvme abort.  We might have to live with this for
> now, but can you please review might addition of new status codes
> for these host-internal cases in the latest ANA draft?

Ok.  Two comments:
1) The same statement must apply to nvme_cancel_request(), which is used 
whenever a controller is torn down before reconnect attempts. That's 
where I pulled the value to return from.

2) The command may have indeed reached the controller and have had some 
level of execution. True though, a NVME Abort would never have reached 
the controller to kill it.

Yes. I'll review the ANA draft recommendations. From my perspective, 
this is independent of ANA and I believe the proposal to add a couple of 
generic transport errors to the base spec was the right thing to do.

I'm good if you want me to change this to a transport error rather than 
the ABORTED status.


> 
>> +	/*
>> +	 * Force failures of commands if we're killing the controller
>> +	 * or have an error on a command used to create an new association
>> +	 */
>> +	if (status && (blk_queue_dying(rq->q) ||
>> +			ctrl->ctrl.state == NVME_CTRL_NEW ||
>> +			ctrl->ctrl.state == NVME_CTRL_RECONNECTING))
>> +		status |= cpu_to_le16(NVME_SC_DNR << 1);
> 
> And here we add back the dnr bit based on a few conditions.

yep - in these cases:
a) blk_queue_dying being where the controller is being deleted/torn down 
for good
b) the io was related to controller initial connect or reconnect  (I 
don't believe "the special" designation on the request was sufficient as 
it didn't apply to read_reg cmds that do get/set_property which could 
have failed too).

we do want them to fail back to the callee completely without retry. if 
(a) the controller dies/is removed. if (b) the controller connect 
attempt fails and the reconnect/retry policy kicks in.


> As a nitpick I think the way the conditionals are nested isn't very
> readable, can we split the first line up, please?
> 
> 	 if (status &&
> 	     (blk_queue_dying(rq->q) ||
> 	      ctrl->ctrl.state == NVME_CTRL_NEW ||
> 	      ctrl->ctrl.state == NVME_CTRL_RECONNECTING))
> 		status |= cpu_to_le16(NVME_SC_DNR << 1);
> 

yes


>>   	complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
>> +	if (!complete_rq)
>>   		nvme_end_request(rq, status, result);
>> +	else
>>   		__nvme_fc_final_op_cleanup(rq);
> 
> avoid the unneeded invert and local variable:
> 
> 	if (unlikely(__nvme_fc_fcpop_chk_teardowns(ctrl, op)))
> 		__nvme_fc_final_op_cleanup(rq);
> 	else
> 		nvme_end_request(rq, status, result);
> 

ok

> and while we're at it: the AEN special case above assigns complete_rq,
> but that value will never be read, so the complete_rq can be removed
> entirely.
> 
> Can you split the reset handler bits into a separate patch with
> a separate explanation?

ok




From hare at suse.de  Tue Oct  3 22:42:00 2017
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 4 Oct 2017 07:42:00 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171003115506.GC24650@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
Message-ID: <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>

On 10/03/2017 01:55 PM, Christoph Hellwig wrote:
> On Tue, Oct 03, 2017 at 12:08:26PM +0200, Hannes Reinecke wrote:
>> With this patch dm-multipath will ignore this device, so it's
>> _impossible_ to setup any device-mapper or md device on top of the
>> 'nvme' device.
>> Without this patch md will happily autostart any RAID device on these
>> devices when it find a signature.
>> Or the user can mount it.
>> Or swap can attach it as a swapspace.
>>
>> _AND_ you can do the same with the subsystem device, too, without any
>> indication that anything is amiss.
>>
>> Do we really want this?
> 
> I though we agreed last week to do dynamic claiming.
> 
> That is if someone claims the multipath-device we extend that claim to
> all /dev/nvmeXnY devices, and if someone claims one of the /dev/nvmeXnY
> devices we extent it to the multipath node.  That should solve all of
> the above issues.
> 
Hmm. Not sure how you would be doing that. Who should be doing the
claiming? Typically the claim is done whenever a device is created on
top of the other...

What about an alternative plan: make creation of the subsystem device
fully dynamic.
But if a subsystem device is created it will always claim the underlying
device. Then we can make the creation dependent on the NMIC attribute,
and existing setups would not be affected.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at lst.de  Tue Oct  3 23:15:33 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 4 Oct 2017 08:15:33 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
Message-ID: <20171004061533.GA20131@lst.de>

On Wed, Oct 04, 2017 at 07:42:00AM +0200, Hannes Reinecke wrote:
> Hmm. Not sure how you would be doing that. Who should be doing the
> claiming? Typically the claim is done whenever a device is created on
> top of the other...

We'd need a callback in the driver if it is claimed, and use that
to for propagating the claim, or use a shared struture to record the
claim.  I haven't looked into the details yet, though.

> What about an alternative plan: make creation of the subsystem device
> fully dynamic.
> But if a subsystem device is created it will always claim the underlying
> device. Then we can make the creation dependent on the NMIC attribute,
> and existing setups would not be affected.

This doesn't work because a lot of devices can just set NMIC.  E.g.
every namespace exported by the Linux NVMe target.


From hch at lst.de  Tue Oct  3 23:30:28 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 4 Oct 2017 08:30:28 +0200
Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <20171002172128.GV8463@localhost.localdomain>
References: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
 <20171001074203.GA11115@lst.de> <20171002172128.GV8463@localhost.localdomain>
Message-ID: <20171004063027.GA20459@lst.de>

On Mon, Oct 02, 2017 at 11:21:29AM -0600, Keith Busch wrote:
> Yah, calling this a DMA address was a misnomer and confusing.

Abhishek, can you test if this works for you?


From hare at suse.de  Tue Oct  3 23:33:07 2017
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 4 Oct 2017 08:33:07 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171004061533.GA20131@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
Message-ID: <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>

On 10/04/2017 08:15 AM, Christoph Hellwig wrote:
> On Wed, Oct 04, 2017 at 07:42:00AM +0200, Hannes Reinecke wrote:
>> Hmm. Not sure how you would be doing that. Who should be doing the
>> claiming? Typically the claim is done whenever a device is created on
>> top of the other...
> 
> We'd need a callback in the driver if it is claimed, and use that
> to for propagating the claim, or use a shared struture to record the
> claim.  I haven't looked into the details yet, though.
> 
>> What about an alternative plan: make creation of the subsystem device
>> fully dynamic.
>> But if a subsystem device is created it will always claim the underlying
>> device. Then we can make the creation dependent on the NMIC attribute,
>> and existing setups would not be affected.
> 
> This doesn't work because a lot of devices can just set NMIC.  E.g.
> every namespace exported by the Linux NVMe target.
> 
But as it's fully dynamic we can decide how to handle each device on a
device-by-device basis, with a common default policy.
My idea is to have a default policy (create/not create subsystem devices
if NMIC is set) set via kernel command-line, and udev rules for devices
requiring separate handling.

Will be drafting up a set of patches to demonstrate the idea.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at lst.de  Wed Oct  4 00:13:12 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 4 Oct 2017 09:13:12 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
Message-ID: <20171004071312.GA21143@lst.de>

On Wed, Oct 04, 2017 at 08:33:07AM +0200, Hannes Reinecke wrote:
> > On Wed, Oct 04, 2017 at 07:42:00AM +0200, Hannes Reinecke wrote:
> >> Hmm. Not sure how you would be doing that. Who should be doing the
> >> claiming? Typically the claim is done whenever a device is created on
> >> top of the other...
> > 
> > We'd need a callback in the driver if it is claimed, and use that
> > to for propagating the claim, or use a shared struture to record the
> > claim.  I haven't looked into the details yet, though.
> > 
> >> What about an alternative plan: make creation of the subsystem device
> >> fully dynamic.
> >> But if a subsystem device is created it will always claim the underlying
> >> device. Then we can make the creation dependent on the NMIC attribute,
> >> and existing setups would not be affected.
> > 
> > This doesn't work because a lot of devices can just set NMIC.  E.g.
> > every namespace exported by the Linux NVMe target.
> > 
> But as it's fully dynamic we can decide how to handle each device on a
> device-by-device basis, with a common default policy.

What do you mean with fully dynamic?

> My idea is to have a default policy (create/not create subsystem devices
> if NMIC is set) set via kernel command-line, and udev rules for devices
> requiring separate handling.

Yikes.  That's exactly where I do _not_ want to go.  No more arcane
magic setup that you need to be in the in group for to know like
dm-multipath.  Things must just work, period.

The only other option I could think of would be to turn names around:
make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
that are multipathed if available.  We'd then need new devices for the
invdividual controllers.  With a little luck we'd get away with just
creating character devices.  This would however create a problem for
dm-multipath users, which mostly is you and your paterners/customers.

Note that in general dm-multipath should continue working but it would
generally just see one path.


From hare at suse.de  Wed Oct  4 00:26:56 2017
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 4 Oct 2017 09:26:56 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171004071312.GA21143@lst.de>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de>
Message-ID: <d476287b-63b2-f1ee-ba18-73e55a70da49@suse.de>

On 10/04/2017 09:13 AM, Christoph Hellwig wrote:
> On Wed, Oct 04, 2017 at 08:33:07AM +0200, Hannes Reinecke wrote:
>>> On Wed, Oct 04, 2017 at 07:42:00AM +0200, Hannes Reinecke wrote:
>>>> Hmm. Not sure how you would be doing that. Who should be doing the
>>>> claiming? Typically the claim is done whenever a device is created on
>>>> top of the other...
>>>
>>> We'd need a callback in the driver if it is claimed, and use that
>>> to for propagating the claim, or use a shared struture to record the
>>> claim.  I haven't looked into the details yet, though.
>>>
>>>> What about an alternative plan: make creation of the subsystem device
>>>> fully dynamic.
>>>> But if a subsystem device is created it will always claim the underlying
>>>> device. Then we can make the creation dependent on the NMIC attribute,
>>>> and existing setups would not be affected.
>>>
>>> This doesn't work because a lot of devices can just set NMIC.  E.g.
>>> every namespace exported by the Linux NVMe target.
>>>
>> But as it's fully dynamic we can decide how to handle each device on a
>> device-by-device basis, with a common default policy.
> 
> What do you mean with fully dynamic?
> 
>> My idea is to have a default policy (create/not create subsystem devices
>> if NMIC is set) set via kernel command-line, and udev rules for devices
>> requiring separate handling.
> 
> Yikes.  That's exactly where I do _not_ want to go.  No more arcane
> magic setup that you need to be in the in group for to know like
> dm-multipath.  Things must just work, period.
> 
:-)
I knew you'd be answering that.
I do agree that we need to make the setup as simple as possible.
But at the same time we should avoid the mistakes we've made with
dm-multipathing.

> The only other option I could think of would be to turn names around:
> make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
> that are multipathed if available.  We'd then need new devices for the
> invdividual controllers.  With a little luck we'd get away with just
> creating character devices.  This would however create a problem for
> dm-multipath users, which mostly is you and your paterners/customers.
> 
_That_ idea is by far the best approach I've seen so far.
Which would essentially move the entire path hierarchy _below_ the nvme
device, and we would always have a single device node to speak with.
Which will decrease the maintenance burden for us poor souls having to
make a distribution out of it.

_Plus_ we can rearrange things underneath it even when mounted, so with
this approach it would be possible to move from non-multipath to
multipath and vice versa.

So go for it.

> Note that in general dm-multipath should continue working but it would
> generally just see one path.
> 
Yeah, to be expected. But I guess I can find ways around this.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at infradead.org  Wed Oct  4 00:44:19 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 00:44:19 -0700
Subject: [PATCH] nvme: update timeout module parameter type
In-Reply-To: <1504743836-25516-1-git-send-email-marcolso@amazon.com>
References: <1504743836-25516-1-git-send-email-marcolso@amazon.com>
Message-ID: <20171004074419.GA30264@infradead.org>

Thanks,

applied to nvme-4.15.


From hch at infradead.org  Wed Oct  4 00:46:38 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 09:46:38 +0200
Subject: [PATCH] nvmet: bump NVMET_NR_QUEUES to 128
In-Reply-To: <20170911231450.20409-1-jsmart2021@gmail.com>
References: <20170911231450.20409-1-jsmart2021@gmail.com>
Message-ID: <20171004074638.qfw3qyowslc553xo@infradead.org>

Thanks, applied to nvme-4.15.


From hch at infradead.org  Wed Oct  4 00:47:03 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 09:47:03 +0200
Subject: [PATCH] nvme: use menu Kconfig interface
In-Reply-To: <1c39a10d-6782-3737-80f7-e29fe0d12980@infradead.org>
References: <1c39a10d-6782-3737-80f7-e29fe0d12980@infradead.org>
Message-ID: <20171004074703.uvn23abckc5nnjmb@infradead.org>

Thanks, applied to nvme-4.15.


From hch at infradead.org  Wed Oct  4 00:47:22 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 09:47:22 +0200
Subject: [PATCH v2] nvme-fabrics: request transport module
In-Reply-To: <1506258955-18109-1-git-send-email-sagi@grimberg.me>
References: <1506258955-18109-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171004074722.pwlgzvwpyohilxoy@infradead.org>

Applied to nvme-4.15.


From hch at infradead.org  Wed Oct  4 00:48:49 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 09:48:49 +0200
Subject: [PATCH v3 0/3] nvme_fc: add uevent to allow dynamic connects
In-Reply-To: <20170914173843.3195-1-jsmart2021@gmail.com>
References: <20170914173843.3195-1-jsmart2021@gmail.com>
Message-ID: <20171004074849.wphq65tudvtk4g6d@infradead.org>

Applied patches 1 and 2 to nvme-4.15.


From hch at infradead.org  Wed Oct  4 01:28:13 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 4 Oct 2017 01:28:13 -0700
Subject: [PATCH v2 1/7] nvme core: allow controller RESETTING to
 RECONNECTING transition
In-Reply-To: <20170927045046.22238-2-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-2-jsmart2021@gmail.com>
Message-ID: <20171004082813.GA23467@infradead.org>

On Tue, Sep 26, 2017 at 09:50:40PM -0700, James Smart wrote:
> Allow controller state transition : RESETTING to RECONNECTING
> 
> Transport will typically transition from LIVE->RESETTING when
> initially performing a reset or recovering from an error. Adding
> this transition allows a transport to transition to RECONNECTING
> when it checks/waits for connectivity then creates new transport
> connections and reinits the controller.

Looks fine for now, although we really need to move to a common
state machine..

Reviewed-by: Christoph Hellwig <hch at lst.de>

> 
> -- james
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> ---
>  drivers/nvme/host/core.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index bb2aad078637..179ca6f79da3 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -205,6 +205,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
>  	case NVME_CTRL_RECONNECTING:
>  		switch (old_state) {
>  		case NVME_CTRL_LIVE:
> +		case NVME_CTRL_RESETTING:
>  			changed = true;
>  			/* FALLTHRU */
>  		default:
> -- 
> 2.13.1
> 
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
---end quoted text---


From abhishek.shah at broadcom.com  Wed Oct  4 02:37:17 2017
From: abhishek.shah at broadcom.com (Abhishek Shah)
Date: Wed, 4 Oct 2017 15:07:17 +0530
Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <20171004063027.GA20459@lst.de>
References: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
 <20171001074203.GA11115@lst.de> <20171002172128.GV8463@localhost.localdomain>
 <20171004063027.GA20459@lst.de>
Message-ID: <CAKUFe6aqX5V10eszHMaD906+zhvk9iavOKSn0t0=XYM1rA9_3w@mail.gmail.com>

yes, this patch works for our platform.

On Wed, Oct 4, 2017 at 12:00 PM, Christoph Hellwig <hch at lst.de> wrote:
> On Mon, Oct 02, 2017 at 11:21:29AM -0600, Keith Busch wrote:
>> Yah, calling this a DMA address was a misnomer and confusing.
>
> Abhishek, can you test if this works for you?


From mwilck at suse.com  Wed Oct  4 03:33:12 2017
From: mwilck at suse.com (Martin Wilck)
Date: Wed, 04 Oct 2017 12:33:12 +0200
Subject: [PATCH 1/2] block: genhd: add device_add_disk_with_groups
In-Reply-To: <20171001080003.GD11554@lst.de>
References: <20170928193637.24707-1-mwilck@suse.com>
 <20171001080003.GD11554@lst.de>
Message-ID: <1507113192.22324.1.camel@suse.com>

On Sun, 2017-10-01 at 10:00 +0200, Christoph Hellwig wrote:
> While this looks okay-ish to me I really don't want people confused
> with three variants of add_disk, we really need to consolidate
> our helpers there a bit..
> 

Can you give me a hint what you'd like to see?

Martin

-- 
Dr. Martin Wilck <mwilck at suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)



From mwilck at suse.com  Wed Oct  4 03:46:42 2017
From: mwilck at suse.com (Martin Wilck)
Date: Wed, 04 Oct 2017 12:46:42 +0200
Subject: [PATCH 1/2] block: genhd: add device_add_disk_with_groups
In-Reply-To: <20170929225949.GP8463@localhost.localdomain>
References: <20170928193637.24707-1-mwilck@suse.com>
 <20170929225949.GP8463@localhost.localdomain>
Message-ID: <1507114002.22324.12.camel@suse.com>

On Fri, 2017-09-29 at 16:59 -0600, Keith Busch wrote:
> On Thu, Sep 28, 2017 at 09:36:36PM +0200, Martin Wilck wrote:
> > In the NVME subsystem, we're seeing a race condition with udev
> > where
> > device_add_disk() is called (which triggers an "add" uevent), and a
> > sysfs attribute group is added to the disk device afterwards.
> > If udev rules access these attributes before they are created,
> > udev processing of the device is incomplete, in particular, device
> > WWIDs may not be determined correctly.
> > 
> > To fix this, this patch introduces a new function
> > device_add_disk_with_groups(), which takes a list of attribute
> > groups
> > and adds them to the device before sending out uevents.
> > 
> > Signed-off-by: Martin Wilck <mwilck at suse.com>
> 
> Is NVMe the only one having this problem?

There are other devices that follow the same programming pattern
(device_add_disk followed by sysfs_create_group), but I haven't tested
them all, nor reviewed whether these devices need the disk's sysfs
attributes for udev processing. If they don't, the problem will just go
unnoticed.

SCSI obviously takes a very different approach to sysfs layout.

> Was putting our attributes in the disk's kobj a bad choice?

Well, it would make sense to separate the block (disk) device from the
device representing the NVMe subsys/namespace in sysfs. But I guess
it's too late for that now. Actually, the first attempt I made to solve
this problem was exactly that, and it proved to "work", too, but only
at the cost of changing the path of the NVMe block device in sysfs,
which I considered a no-go. Thus I came up with the approach I posted.

Regards,
Martin

-- 
Dr. Martin Wilck <mwilck at suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)



From nnlnachu at gmail.com  Wed Oct  4 05:13:05 2017
From: nnlnachu at gmail.com (Nachammai Lakshmanan)
Date: Wed, 4 Oct 2017 17:43:05 +0530
Subject: A way in nvme driver to control the maximum queue depth
Message-ID: <CAEy7waaGj01g0eXG58e4b8_rDpg23ZW9puWZzCBe1vWT=v1gHA@mail.gmail.com>

Hi,



I am looking for an ioctl to enable queue depth greater than 1.
Something similar to SG_SET_COMMAND_Q (which is for SCSI devices).

And also is there a way to get/set the max queue depth at driver layer
? (For, SCSI device this can be done by using SG_GET_SCSI_ID ? where
sg_scsi_id structure has a field(d_queue_depth) to get/set queue
depth)



Regards,

Nachammai


From keith.busch at intel.com  Wed Oct  4 13:04:00 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 4 Oct 2017 14:04:00 -0600
Subject: [PATCH V4 1/2] nvme: add SGL controller capability constant
In-Reply-To: <1507058666-15298-2-git-send-email-ckulkarnilinux@gmail.com>
References: <1507058666-15298-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507058666-15298-2-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <20171004200359.GA2384@localhost.localdomain>

On Tue, Oct 03, 2017 at 12:24:25PM -0700, Chaitanya Kulkarni wrote:
> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> diff --git a/include/linux/nvme.h b/include/linux/nvme.h
> index 9310ce77..22bbcfb 100644
> --- a/include/linux/nvme.h
> +++ b/include/linux/nvme.h
> @@ -466,6 +466,8 @@ enum nvme_opcode {
>  	nvme_cmd_resv_release	= 0x15,
>  };
>  
> +#define NVME_CTRL_SUPP_SGL     0x1
> +

I don't think this needs to be its own patch.

This should be in enum along with all the other NVME_CTRL_* identify
controller constants.

SGLs would also be supported if the value is 2 as well, right?


From ckulkarnilinux at gmail.com  Wed Oct  4 17:55:17 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Wed,  4 Oct 2017 17:55:17 -0700
Subject: [PATCH V5] nvme-pci: add SGL support
Message-ID: <1507164918-4749-1-git-send-email-ckulkarnilinux@gmail.com>

This is the fifth version of the patch for adding SGL support
for nvme-pci driver.

Changes since v4:-

1. Replace NVME_CTRL_SUPP_SGL macro with enum NVME_CTRL_SGL_NOT_SUPP.
2. Adjust patch for new constant.
3. Merge two patches into one.

Changes since v3:-

1. Modify SGL helper functions and add prefix nvme_pci_sgl.
2. Use average request physical segment(s) size to conditionally use SGLs.
   This approach is different from the original one, the original approach
   was based no the IO size. The new approach considers IO size and 
   number of physical segments present in the request to conditionally
   use SGLs.
3. Adjust the patch for Linux 4.14-rc3.

Changes since v2:-

1. Adjust the patch for latest nvme driver changes.
2. For initial submission use the default virt boundary values.
    
In future, work on the patch series for relaxing the virt boundary
for each transport.

[PATCH V3] nvme-pci: add sgl support

Changes since the v1:-

1. Introduced nvme controller virt_boundary mask. For controllers that
   can handle arbitrarily sized bios (e.g advanced RDMA and PCI ctrls)
   we can allow the block layer to pass us gaps by setting the 
   queue.virt_boundary_mask according to each ctrl constraints.
2. Simplified code for setting up the SGLs.
3. Added SGL helper functions.
4. Modified commit log.
5. Use the newly introduced virt_boundary mask and clear its 
   value for SGL capable controllers in PCIe driver.

For reference v1 details:-

This adds SGL support for NVMe PCIe driver which is
reimplementation of the initial version provided by
Rajiv Shanmugam Madeswaran(smrajiv15 at gmail.com):-
"block: nvme-core: Scatter gather list support in the 
NVMe Block Driver".

In order to enable SGL mode for the driver, the user can set the 
sgl_threshold module parameter. This parameter can be used
in the two modes:-

1. Allow all IOs to use SGL mode. (set sgl_threshold to 1). 
2. Set up the IO size threshold to determine whether to use 
    SGLs or PRPs for each IO. (set sgl_threshold to 4096 to
    use SGL only for IOs which are >= 4096 in the size).


-Regards,
Chaitanya


From ckulkarnilinux at gmail.com  Wed Oct  4 17:55:18 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Wed,  4 Oct 2017 17:55:18 -0700
Subject: [PATCH V5] nvme-pci: add SGL support
In-Reply-To: <1507164918-4749-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1507164918-4749-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1507164918-4749-2-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds SGL support for NVMe PCIe driver, based on an earlier patch
from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
refactors the original code and adds new module parameter sgl_threshold
to determine whether to use SGL or PRP for IOs.

The new enum controller capability constant NVME_SGL_NOT_SUPP
definition is used to determine whether NVMe Controller supports SGL
mode for IOs.

The usage of SGLs is controlled by the sgl_threshold module parameter,
which allows to conditionally use SGLs if average request segment size
(avg_seg_size) is greater than sgl_threshold. In the original patch,
the decision of using SGLs was dependent only on the IO size,
with the new approach we consider not only IO size but also the
number of physical segments present in the IO.

We calculate avg_seg_size based on request payload bytes and number
of physical segments present in the request.

For e.g.:-

1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.

3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
avg_seg_size = 4K use sgl if avg_seg_size

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 223 ++++++++++++++++++++++++++++++++++++++++++------
 include/linux/nvme.h    |   1 +
 2 files changed, 199 insertions(+), 25 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cb73bc8..9f66f75 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -45,6 +45,8 @@
  */
 #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
 
+#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+
 static int use_threaded_interrupts;
 module_param(use_threaded_interrupts, int, 0);
 
@@ -57,6 +59,10 @@
 MODULE_PARM_DESC(max_host_mem_size_mb,
 	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
 
+static unsigned int sgl_threshold = SZ_32K;
+module_param(sgl_threshold, uint, 0644);
+MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this size");
+
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
 static const struct kernel_param_ops io_queue_depth_ops = {
 	.set = io_queue_depth_set,
@@ -178,6 +184,7 @@ struct nvme_queue {
 struct nvme_iod {
 	struct nvme_request req;
 	struct nvme_queue *nvmeq;
+	bool use_sgl;
 	int aborted;
 	int npages;		/* In the PRP list. 0 means small pool in use */
 	int nents;		/* Used in scatterlist */
@@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
 	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
 }
 
+/*
+ * Calculates the number of pages needed for the SGL segments. For example a 4k
+ * page can accommodate 256 SGL descriptors.
+ */
+static int nvme_npages_sgl(unsigned int num_seg)
+{
+	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
+}
+
 static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
-		unsigned int size, unsigned int nseg)
+		unsigned int size, unsigned int nseg, bool use_sgl)
 {
-	return sizeof(__le64 *) * nvme_npages(size, dev) +
-			sizeof(struct scatterlist) * nseg;
+	size_t alloc_size;
+
+	if (use_sgl)
+		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
+	else
+		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
+
+	return alloc_size + sizeof(struct scatterlist) * nseg;
 }
 
-static unsigned int nvme_cmd_size(struct nvme_dev *dev)
+static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
 {
 	return sizeof(struct nvme_iod) +
-		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
+		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
 }
 
 static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
@@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
 	nvmeq->sq_tail = tail;
 }
 
-static __le64 **iod_list(struct request *req)
+static void **iod_list(struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
+	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
 }
 
 static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
@@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 	unsigned int size = blk_rq_payload_bytes(rq);
 
 	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
-		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
+		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
+				iod->use_sgl);
+
+		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
 		if (!iod->sg)
 			return BLK_STS_RESOURCE;
 	} else {
@@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	const int last_prp = dev->ctrl.page_size / 8 - 1;
+	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
+	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
 	int i;
-	__le64 **list = iod_list(req);
-	dma_addr_t prp_dma = iod->first_dma;
 
 	if (iod->npages == 0)
-		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
+		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
+
 	for (i = 0; i < iod->npages; i++) {
-		__le64 *prp_list = list[i];
-		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
-		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
-		prp_dma = next_prp_dma;
+		void *addr = iod_list(req)[i];
+
+		if (iod->use_sgl) {
+			struct nvme_sgl_desc *sg_list = addr;
+
+			next_dma_addr =
+				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
+		} else {
+			__le64 *prp_list = addr;
+
+			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
+		}
+
+		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
+		dma_addr = next_dma_addr;
 	}
 
 	if (iod->sg != iod->inline_sg)
@@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
 	}
 }
 
-static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
+static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmnd)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
 	struct dma_pool *pool;
@@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	u32 page_size = dev->ctrl.page_size;
 	int offset = dma_addr & (page_size - 1);
 	__le64 *prp_list;
-	__le64 **list = iod_list(req);
+	void **list = iod_list(req);
 	dma_addr_t prp_dma;
 	int nprps, i;
 
+	iod->use_sgl = false;
+
 	length -= (page_size - offset);
 	if (length <= 0) {
 		iod->first_dma = 0;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	dma_len -= (page_size - offset);
@@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 
 	if (length <= page_size) {
 		iod->first_dma = dma_addr;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	nprps = DIV_ROUND_UP(length, page_size);
@@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 		dma_len = sg_dma_len(sg);
 	}
 
+done:
+	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
+	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
+
 	return BLK_STS_OK;
 
  bad_sgl:
@@ -643,6 +686,130 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	return BLK_STS_IOERR;
 }
 
+static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
+		struct scatterlist *sg)
+{
+	sge->addr = cpu_to_le64(sg_dma_address(sg));
+	sge->length = cpu_to_le32(sg_dma_len(sg));
+	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr, int entries)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
+	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(PAGE_SIZE);
+	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
+}
+
+static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmd)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	int length = blk_rq_payload_bytes(req);
+	struct dma_pool *pool;
+	struct nvme_sgl_desc *sg_list;
+	struct scatterlist *sg = iod->sg;
+	int entries = iod->nents, i = 0;
+	dma_addr_t sgl_dma;
+
+	iod->use_sgl = true;
+
+	cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as SGL */
+
+	if (length == sg_dma_len(sg)) {
+		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
+		return BLK_STS_OK;
+	}
+
+	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
+		pool = dev->prp_small_pool;
+		iod->npages = 0;
+	} else {
+		pool = dev->prp_page_pool;
+		iod->npages = 1;
+	}
+
+	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+	if (!sg_list) {
+		iod->npages = -1;
+		return BLK_STS_RESOURCE;
+	}
+
+	iod_list(req)[0] = sg_list;
+	iod->first_dma = sgl_dma;
+
+	if (entries <= SGES_PER_PAGE) {
+		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
+
+		for (i = 0; i < entries; i++) {
+			nvme_pci_sgl_set_data(&sg_list[i], sg);
+			length -= sg_dma_len(sg);
+			sg = sg_next(sg);
+		}
+
+		WARN_ON(length > 0);
+		return BLK_STS_OK;
+	}
+
+	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
+
+	do {
+		if (i == SGES_PER_PAGE) {
+			struct nvme_sgl_desc *old_sg_desc = sg_list;
+			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
+
+			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+			if (!sg_list)
+				return BLK_STS_RESOURCE;
+
+			i = 0;
+			iod_list(req)[iod->npages++] = sg_list;
+			sg_list[i++] = *link;
+
+			if (entries < SGES_PER_PAGE)
+				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
+			else
+				nvme_pci_sgl_set_seg(link, sgl_dma);
+		}
+
+		nvme_pci_sgl_set_data(&sg_list[i], sg);
+
+		length -= sg_dma_len(sg);
+		sg = sg_next(sg);
+		entries--;
+	} while (length > 0);
+
+	WARN_ON(entries > 0);
+	return BLK_STS_OK;
+}
+
+static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	unsigned int avg_seg_size;
+
+	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
+			blk_rq_nr_phys_segments(req));
+
+	if (dev->ctrl.sgls == NVME_CTRL_SGL_NOT_SUPP)
+		return false;
+	if (!iod->nvmeq->qid)
+		return false;
+	if (!sgl_threshold || avg_seg_size < sgl_threshold)
+		return false;
+	return true;
+}
+
+
 static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 		struct nvme_command *cmnd)
 {
@@ -662,7 +829,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 				DMA_ATTR_NO_WARN))
 		goto out;
 
-	ret = nvme_setup_prps(dev, req);
+	if (nvme_pci_use_sgls(dev, req))
+		ret = nvme_setup_sgls(dev, req, &cmnd->rw);
+	else
+		ret = nvme_setup_prps(dev, req, &cmnd->rw);
+
 	if (ret != BLK_STS_OK)
 		goto out_unmap;
 
@@ -682,8 +853,6 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 			goto out_unmap;
 	}
 
-	cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
-	cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
 	if (blk_integrity_rq(req))
 		cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
 	return BLK_STS_OK;
@@ -1379,7 +1548,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
-		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
+		dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
 		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
 		dev->admin_tagset.driver_data = dev;
 
@@ -1906,7 +2075,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
 		dev->tagset.numa_node = dev_to_node(dev->dev);
 		dev->tagset.queue_depth =
 				min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
-		dev->tagset.cmd_size = nvme_cmd_size(dev);
+		dev->tagset.cmd_size = nvme_cmd_size(dev, false);
+		if ((dev->ctrl.sgls != NVME_CTRL_SGL_NOT_SUPP) && sgl_threshold) {
+			dev->tagset.cmd_size = max(dev->tagset.cmd_size,
+					nvme_cmd_size(dev, true));
+		}
 		dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
 		dev->tagset.driver_data = dev;
 
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9310ce77..bb7e155 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -267,6 +267,7 @@ enum {
 	NVME_CTRL_OACS_SEC_SUPP                 = 1 << 0,
 	NVME_CTRL_OACS_DIRECTIVES		= 1 << 5,
 	NVME_CTRL_OACS_DBBUF_SUPP		= 1 << 8,
+	NVME_CTRL_SGL_NOT_SUPP			= 0,
 };
 
 struct nvme_lbaf {
-- 
1.8.3.1



From martin.petersen at oracle.com  Wed Oct  4 18:08:57 2017
From: martin.petersen at oracle.com (Martin K. Petersen)
Date: Wed, 04 Oct 2017 21:08:57 -0400
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171004071312.GA21143@lst.de> (Christoph Hellwig's message of
 "Wed, 4 Oct 2017 09:13:12 +0200")
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de>
Message-ID: <yq1poa2wgd2.fsf@oracle.com>


Christoph,

> The only other option I could think of would be to turn names around:
> make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
> that are multipathed if available.  We'd then need new devices for the
> invdividual controllers.

I like that approach.

-- 
Martin K. Petersen	Oracle Linux Engineering


From jthumshirn at suse.de  Wed Oct  4 23:51:24 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 5 Oct 2017 08:51:24 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <yq1poa2wgd2.fsf@oracle.com>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de> <yq1poa2wgd2.fsf@oracle.com>
Message-ID: <20171005065124.h5t3gtufptevja4s@linux-x5ow.site>

On Wed, Oct 04, 2017 at 09:08:57PM -0400,  Martin K. Petersen  wrote:
> 
> Christoph,
> 
> > The only other option I could think of would be to turn names around:
> > make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
> > that are multipathed if available.  We'd then need new devices for the
> > invdividual controllers.
> 
> I like that approach.

/me too

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at infradead.org  Thu Oct  5 00:52:28 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 00:52:28 -0700
Subject: [PATCH V5] nvme-pci: add SGL support
In-Reply-To: <1507164918-4749-2-git-send-email-ckulkarnilinux@gmail.com>
References: <1507164918-4749-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507164918-4749-2-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <20171005075228.GA25401@infradead.org>

On Wed, Oct 04, 2017 at 05:55:18PM -0700, Chaitanya Kulkarni wrote:
> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> 
> This adds SGL support for NVMe PCIe driver, based on an earlier patch
> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
> refactors the original code and adds new module parameter sgl_threshold
> to determine whether to use SGL or PRP for IOs.
> 
> The new enum controller capability constant NVME_SGL_NOT_SUPP
> definition is used to determine whether NVMe Controller supports SGL
> mode for IOs.
> 
> The usage of SGLs is controlled by the sgl_threshold module parameter,
> which allows to conditionally use SGLs if average request segment size
> (avg_seg_size) is greater than sgl_threshold. In the original patch,
> the decision of using SGLs was dependent only on the IO size,
> with the new approach we consider not only IO size but also the
> number of physical segments present in the IO.
> 
> We calculate avg_seg_size based on request payload bytes and number
> of physical segments present in the request.
> 
> For e.g.:-
> 
> 1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
> avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.
> 
> 2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
> avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.
> 
> 3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
> avg_seg_size = 4K use sgl if avg_seg_size
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/host/pci.c | 223 ++++++++++++++++++++++++++++++++++++++++++------
>  include/linux/nvme.h    |   1 +
>  2 files changed, 199 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index cb73bc8..9f66f75 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -45,6 +45,8 @@
>   */
>  #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
>  
> +#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
> +
>  static int use_threaded_interrupts;
>  module_param(use_threaded_interrupts, int, 0);
>  
> @@ -57,6 +59,10 @@
>  MODULE_PARM_DESC(max_host_mem_size_mb,
>  	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
>  
> +static unsigned int sgl_threshold = SZ_32K;
> +module_param(sgl_threshold, uint, 0644);
> +MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this size");
> +
>  static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
>  static const struct kernel_param_ops io_queue_depth_ops = {
>  	.set = io_queue_depth_set,
> @@ -178,6 +184,7 @@ struct nvme_queue {
>  struct nvme_iod {
>  	struct nvme_request req;
>  	struct nvme_queue *nvmeq;
> +	bool use_sgl;
>  	int aborted;
>  	int npages;		/* In the PRP list. 0 means small pool in use */
>  	int nents;		/* Used in scatterlist */
> @@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
>  	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
>  }
>  
> +/*
> + * Calculates the number of pages needed for the SGL segments. For example a 4k
> + * page can accommodate 256 SGL descriptors.
> + */
> +static int nvme_npages_sgl(unsigned int num_seg)
> +{
> +	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
> +}
> +
>  static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
> -		unsigned int size, unsigned int nseg)
> +		unsigned int size, unsigned int nseg, bool use_sgl)
>  {
> -	return sizeof(__le64 *) * nvme_npages(size, dev) +
> -			sizeof(struct scatterlist) * nseg;
> +	size_t alloc_size;
> +
> +	if (use_sgl)
> +		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
> +	else
> +		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
> +
> +	return alloc_size + sizeof(struct scatterlist) * nseg;
>  }
>  
> -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>  {
>  	return sizeof(struct nvme_iod) +
> -		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
> +		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
>  }
>  
>  static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
>  	nvmeq->sq_tail = tail;
>  }
>  
> -static __le64 **iod_list(struct request *req)
> +static void **iod_list(struct request *req)
>  {
>  	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
> +	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>  }
>  
>  static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>  	unsigned int size = blk_rq_payload_bytes(rq);
>  
>  	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
> -		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
> +		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
> +				iod->use_sgl);
> +
> +		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>  		if (!iod->sg)
>  			return BLK_STS_RESOURCE;
>  	} else {
> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>  static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>  {
>  	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	const int last_prp = dev->ctrl.page_size / 8 - 1;
> +	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
> +	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>  	int i;
> -	__le64 **list = iod_list(req);
> -	dma_addr_t prp_dma = iod->first_dma;
>  
>  	if (iod->npages == 0)
> -		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
> +		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
> +
>  	for (i = 0; i < iod->npages; i++) {
> -		__le64 *prp_list = list[i];
> -		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
> -		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
> -		prp_dma = next_prp_dma;
> +		void *addr = iod_list(req)[i];
> +
> +		if (iod->use_sgl) {
> +			struct nvme_sgl_desc *sg_list = addr;
> +
> +			next_dma_addr =
> +				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
> +		} else {
> +			__le64 *prp_list = addr;
> +
> +			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
> +		}
> +
> +		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
> +		dma_addr = next_dma_addr;
>  	}
>  
>  	if (iod->sg != iod->inline_sg)
> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
>  	}
>  }
>  
> -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmnd)
>  {
>  	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>  	struct dma_pool *pool;
> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>  	u32 page_size = dev->ctrl.page_size;
>  	int offset = dma_addr & (page_size - 1);
>  	__le64 *prp_list;
> -	__le64 **list = iod_list(req);
> +	void **list = iod_list(req);
>  	dma_addr_t prp_dma;
>  	int nprps, i;
>  
> +	iod->use_sgl = false;
> +
>  	length -= (page_size - offset);
>  	if (length <= 0) {
>  		iod->first_dma = 0;
> -		return BLK_STS_OK;
> +		goto done;
>  	}
>  
>  	dma_len -= (page_size - offset);
> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>  
>  	if (length <= page_size) {
>  		iod->first_dma = dma_addr;
> -		return BLK_STS_OK;
> +		goto done;
>  	}
>  
>  	nprps = DIV_ROUND_UP(length, page_size);
> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>  		dma_len = sg_dma_len(sg);
>  	}
>  
> +done:
> +	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
> +	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
> +
>  	return BLK_STS_OK;
>  
>   bad_sgl:
> @@ -643,6 +686,130 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>  	return BLK_STS_IOERR;
>  }
>  
> +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
> +		struct scatterlist *sg)
> +{
> +	sge->addr = cpu_to_le64(sg_dma_address(sg));
> +	sge->length = cpu_to_le32(sg_dma_len(sg));
> +	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr, int entries)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
> +	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(PAGE_SIZE);
> +	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
> +}
> +
> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmd)
> +{
> +	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> +	int length = blk_rq_payload_bytes(req);
> +	struct dma_pool *pool;
> +	struct nvme_sgl_desc *sg_list;
> +	struct scatterlist *sg = iod->sg;
> +	int entries = iod->nents, i = 0;
> +	dma_addr_t sgl_dma;
> +
> +	iod->use_sgl = true;
> +
> +	cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as SGL */
> +
> +	if (length == sg_dma_len(sg)) {
> +		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
> +		return BLK_STS_OK;
> +	}
> +
> +	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
> +		pool = dev->prp_small_pool;
> +		iod->npages = 0;
> +	} else {
> +		pool = dev->prp_page_pool;
> +		iod->npages = 1;
> +	}
> +
> +	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
> +	if (!sg_list) {
> +		iod->npages = -1;
> +		return BLK_STS_RESOURCE;
> +	}
> +
> +	iod_list(req)[0] = sg_list;
> +	iod->first_dma = sgl_dma;
> +
> +	if (entries <= SGES_PER_PAGE) {
> +		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
> +
> +		for (i = 0; i < entries; i++) {
> +			nvme_pci_sgl_set_data(&sg_list[i], sg);
> +			length -= sg_dma_len(sg);
> +			sg = sg_next(sg);
> +		}
> +
> +		WARN_ON(length > 0);
> +		return BLK_STS_OK;
> +	}
> +
> +	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
> +
> +	do {
> +		if (i == SGES_PER_PAGE) {
> +			struct nvme_sgl_desc *old_sg_desc = sg_list;
> +			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
> +
> +			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
> +			if (!sg_list)
> +				return BLK_STS_RESOURCE;
> +
> +			i = 0;
> +			iod_list(req)[iod->npages++] = sg_list;
> +			sg_list[i++] = *link;
> +
> +			if (entries < SGES_PER_PAGE)
> +				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
> +			else
> +				nvme_pci_sgl_set_seg(link, sgl_dma);
> +		}
> +
> +		nvme_pci_sgl_set_data(&sg_list[i], sg);
> +
> +		length -= sg_dma_len(sg);
> +		sg = sg_next(sg);
> +		entries--;
> +	} while (length > 0);
> +
> +	WARN_ON(entries > 0);
> +	return BLK_STS_OK;
> +}
> +
> +static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
> +{
> +	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> +	unsigned int avg_seg_size;
> +
> +	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
> +			blk_rq_nr_phys_segments(req));
> +
> +	if (dev->ctrl.sgls == NVME_CTRL_SGL_NOT_SUPP)
> +		return false;

I'm alsmost tempted to drop the symbolic NVME_CTRL_SGL_NOT_SUPP name
and just opencode something like:

	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1)))
		return false;

We use the raw bit values in various other places, and given how they
appear as such raw bit values in the spec that seems to be easier to
map to me than inventing some artifical name for them.


From hch at infradead.org  Thu Oct  5 00:52:49 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 00:52:49 -0700
Subject: [PATCH] nvmet: bump NVMET_NR_QUEUES to 128
In-Reply-To: <ee9de017-69e9-670e-b763-1eea73598d66@gmail.com>
References: <20170911231450.20409-1-jsmart2021@gmail.com>
 <20170914170300.GA1235@infradead.org>
 <ee9de017-69e9-670e-b763-1eea73598d66@gmail.com>
Message-ID: <20171005075249.GB25401@infradead.org>

On Thu, Sep 14, 2017 at 11:36:48AM -0700, James Smart wrote:
> On 9/14/2017 10:03 AM, Christoph Hellwig wrote:
> > This looks ok as a quick check, but any chance you could send
> > a patch to make this configurable through configfs?  Preferably
> > also for the max queue size.
> 
> Yep - I'll put one together

Did you get a chance to look at that?


From hch at infradead.org  Thu Oct  5 00:57:30 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 00:57:30 -0700
Subject: [PATCH v2 2/7] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20170927045046.22238-3-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-3-jsmart2021@gmail.com>
Message-ID: <20171005075730.GA5063@infradead.org>

>  	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
> -	WARN_ON_ONCE(!changed);
>  
>  	ctrl->ctrl.nr_reconnects = 0;
>  
> -	nvme_start_ctrl(&ctrl->ctrl);
> +	if (changed)
> +		nvme_start_ctrl(&ctrl->ctrl);

It's just cosmetic, but can you folow the RDMA pattern here:

	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
	if (!changed) {
		/* state change failure is ok if we're in DELETING state */
		WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
		return;
	}

else the patch looks fine:

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at infradead.org  Thu Oct  5 01:00:40 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 01:00:40 -0700
Subject: [PATCH v2 4/7] nvme_fc: add dev_loss_tmo to controller
In-Reply-To: <20170927045046.22238-5-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-5-jsmart2021@gmail.com>
Message-ID: <20171005080040.GB5063@infradead.org>

Can you please add the dev_loss_tmo to the generic nvme_ctrl and build
the infrastructure in common code?

Then as a next step we can look into inferring it from the remote port
for FC.


From hch at infradead.org  Thu Oct  5 01:01:39 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 01:01:39 -0700
Subject: [PATCH v2 5/7] nvme_fc: check connectivity before initiating
 reconnects
In-Reply-To: <20170927045046.22238-6-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-6-jsmart2021@gmail.com>
Message-ID: <20171005080139.GC5063@infradead.org>

> +nvme_fc_rport_is_online(struct nvme_fc_rport *rport)
> +{
> +	unsigned long flags;
> +	bool online;
> +
> +	spin_lock_irqsave(&rport->lock, flags);
> +	online = (rport->remoteport.port_state == FC_OBJSTATE_ONLINE);
> +	spin_unlock_irqrestore(&rport->lock, flags);
> +
> +	return online;

There is no need to take a lock for reading a value <= the native
register size ever.

So just open code the check at the callsites, and with that the
patch looks fine to me.


From hch at infradead.org  Thu Oct  5 01:03:20 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 01:03:20 -0700
Subject: [PATCH v2 6/7] nvme_fc: move remote port get/put/free location
In-Reply-To: <20170927045046.22238-7-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-7-jsmart2021@gmail.com>
Message-ID: <20171005080320.GD5063@infradead.org>

On Tue, Sep 26, 2017 at 09:50:45PM -0700, James Smart wrote:
> move nvme_fc_rport_get/put and rport free to higher in the file to
> avoid adding prototypes to resolve references in upcoming code additions
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>

Looks fine.

I've picked this up in nvme-4.15 as large code movements are bound
to create conflicts, so I'd rather have it earlier than later.


From keith.busch at intel.com  Thu Oct  5 07:05:26 2017
From: keith.busch at intel.com (Keith Busch)
Date: Thu, 5 Oct 2017 08:05:26 -0600
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171005065124.h5t3gtufptevja4s@linux-x5ow.site>
References: <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de> <yq1poa2wgd2.fsf@oracle.com>
 <20171005065124.h5t3gtufptevja4s@linux-x5ow.site>
Message-ID: <20171005140525.GB2384@localhost.localdomain>

On Thu, Oct 05, 2017 at 08:51:24AM +0200, Johannes Thumshirn wrote:
> On Wed, Oct 04, 2017 at 09:08:57PM -0400,  Martin K. Petersen  wrote:
> > 
> > Christoph,
> > 
> > > The only other option I could think of would be to turn names around:
> > > make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
> > > that are multipathed if available.  We'd then need new devices for the
> > > invdividual controllers.
> > 
> > I like that approach.
> 
> /me too

I can get on board with this approach for the namespaces, but the
controller char devs should stay as they are for compatibility and
existing tools. The IO commands (including reservations) are not
controller specific, so not knowing which path is used will okay for
namespaces. Most admin commands are controller specific, so we can't
just shift the controller handle to the subsystem level.


From hare at suse.com  Thu Oct  5 07:39:23 2017
From: hare at suse.com (Hannes Reinecke)
Date: Thu, 5 Oct 2017 16:39:23 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <20171005140525.GB2384@localhost.localdomain>
References: <1506952559-1588-4-git-send-email-hare@suse.de>
 <20171002164218.GA11886@lst.de>
 <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de> <yq1poa2wgd2.fsf@oracle.com>
 <20171005065124.h5t3gtufptevja4s@linux-x5ow.site>
 <20171005140525.GB2384@localhost.localdomain>
Message-ID: <6e9b4ef9-c88a-d8b0-1ae7-16e8403e60df@suse.com>

On 10/05/2017 04:05 PM, Keith Busch wrote:
> On Thu, Oct 05, 2017 at 08:51:24AM +0200, Johannes Thumshirn wrote:
>> On Wed, Oct 04, 2017 at 09:08:57PM -0400,  Martin K. Petersen  wrote:
>>>
>>> Christoph,
>>>
>>>> The only other option I could think of would be to turn names around:
>>>> make /dev/nvmeX (chardev) and /dev/nvmeXnY the per-subsystem devices
>>>> that are multipathed if available.  We'd then need new devices for the
>>>> invdividual controllers.
>>>
>>> I like that approach.
>>
>> /me too
> 
> I can get on board with this approach for the namespaces, but the
> controller char devs should stay as they are for compatibility and
> existing tools. The IO commands (including reservations) are not
> controller specific, so not knowing which path is used will okay for
> namespaces. Most admin commands are controller specific, so we can't
> just shift the controller handle to the subsystem level.
> 
I'm fine with having individual char devs for the controller and the
namespaces. That was the plan anyway, and they don't get into play with
systemd et al.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare at suse.com			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at lst.de  Thu Oct  5 07:47:38 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 5 Oct 2017 16:47:38 +0200
Subject: [PATCH 3/6] nvme: claim block devices
In-Reply-To: <6e9b4ef9-c88a-d8b0-1ae7-16e8403e60df@suse.com>
References: <ece83015-0def-8e36-5645-7be00be95c40@suse.de>
 <20171003115506.GC24650@lst.de>
 <bd388bdf-25ac-8b25-f832-ba1316a64dc4@suse.de>
 <20171004061533.GA20131@lst.de>
 <19dbdf11-86c4-b6ea-a18c-a78fcae822d5@suse.de>
 <20171004071312.GA21143@lst.de> <yq1poa2wgd2.fsf@oracle.com>
 <20171005065124.h5t3gtufptevja4s@linux-x5ow.site>
 <20171005140525.GB2384@localhost.localdomain>
 <6e9b4ef9-c88a-d8b0-1ae7-16e8403e60df@suse.com>
Message-ID: <20171005144738.GA28996@lst.de>

On Thu, Oct 05, 2017 at 04:39:23PM +0200, Hannes Reinecke wrote:
> I'm fine with having individual char devs for the controller and the
> namespaces. That was the plan anyway, and they don't get into play with
> systemd et al.

How hardcoded is the assumption that /dev/nvmeX is the chardev for
/dev/nvmeXn*?

I thought that was quite deep and thus would want to keep /dev/nvmeX
as a multipathed devices (maybe some lightweight multipathing, aka
always go to the first controller) and then have another chardev
like /dev/nvmeXpN or /dev/nvmeXcN for each controller?


From hch at infradead.org  Thu Oct  5 09:36:54 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 5 Oct 2017 18:36:54 +0200
Subject: [GIT PULL] nvme fixes for Linux 4.14-rc4
Message-ID: <20171005163654.walnk7642obfyjp6@infradead.org>

A trivial one-liner from Martin to fix the visible of the uuid attr,
and another one (originally from Abhishek Shah, rewritten by me) to
fix the CMB addresses passed back to the controller in case of a system
that remaps BAR addresses between host and device.

The following changes since commit a5f3d8a5eaaf917878f07998e6f1ea46024e6bab:

  bcache: use llist_for_each_entry_safe() in __closure_wake_up() (2017-09-27 14:54:49 -0600)

are available in the git repository at:

  git://git.infradead.org/nvme.git nvme-4.14

for you to fetch changes up to 8969f1f8291762c13147c1ba89d46238af01675b:

  nvme-pci: Use PCI bus address for data/queues in CMB (2017-10-04 11:42:53 +0200)

----------------------------------------------------------------
Christoph Hellwig (1):
      nvme-pci: Use PCI bus address for data/queues in CMB

Martin Wilck (1):
      nvme: fix visibility of "uuid" ns attribute

 drivers/nvme/host/core.c |  2 +-
 drivers/nvme/host/pci.c  | 14 +++++++-------
 2 files changed, 8 insertions(+), 8 deletions(-)


From axboe at kernel.dk  Thu Oct  5 09:45:12 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Thu, 5 Oct 2017 10:45:12 -0600
Subject: [GIT PULL] nvme fixes for Linux 4.14-rc4
In-Reply-To: <20171005163654.walnk7642obfyjp6@infradead.org>
References: <20171005163654.walnk7642obfyjp6@infradead.org>
Message-ID: <8f1e6bc2-f511-7209-bc28-3876ed806755@kernel.dk>

On 10/05/2017 10:36 AM, Christoph Hellwig wrote:
> A trivial one-liner from Martin to fix the visible of the uuid attr,
> and another one (originally from Abhishek Shah, rewritten by me) to
> fix the CMB addresses passed back to the controller in case of a system
> that remaps BAR addresses between host and device.
> 
> The following changes since commit a5f3d8a5eaaf917878f07998e6f1ea46024e6bab:
> 
>   bcache: use llist_for_each_entry_safe() in __closure_wake_up() (2017-09-27 14:54:49 -0600)
> 
> are available in the git repository at:
> 
>   git://git.infradead.org/nvme.git nvme-4.14
> 
> for you to fetch changes up to 8969f1f8291762c13147c1ba89d46238af01675b:
> 
>   nvme-pci: Use PCI bus address for data/queues in CMB (2017-10-04 11:42:53 +0200)
> 
> ----------------------------------------------------------------
> Christoph Hellwig (1):
>       nvme-pci: Use PCI bus address for data/queues in CMB
> 
> Martin Wilck (1):
>       nvme: fix visibility of "uuid" ns attribute

Pulled, thanks.

-- 
Jens Axboe



From hch at lst.de  Thu Oct  5 09:46:46 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu,  5 Oct 2017 18:46:46 +0200
Subject: [PATCH] nvme: simplify compat_ioctl handling
Message-ID: <20171005164646.17473-1-hch@lst.de>

We can just use our normal ioctl handler for the compat case and remove
the boilerplate code for it.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 26c8913435b2..573cc3b59bfa 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1052,16 +1052,6 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 	}
 }
 
-#ifdef CONFIG_COMPAT
-static int nvme_compat_ioctl(struct block_device *bdev, fmode_t mode,
-			unsigned int cmd, unsigned long arg)
-{
-	return nvme_ioctl(bdev, mode, cmd, arg);
-}
-#else
-#define nvme_compat_ioctl	NULL
-#endif
-
 static int nvme_open(struct block_device *bdev, fmode_t mode)
 {
 	return nvme_get_ns_from_disk(bdev->bd_disk) ? 0 : -ENXIO;
@@ -1380,7 +1370,7 @@ EXPORT_SYMBOL_GPL(nvme_sec_submit);
 static const struct block_device_operations nvme_fops = {
 	.owner		= THIS_MODULE,
 	.ioctl		= nvme_ioctl,
-	.compat_ioctl	= nvme_compat_ioctl,
+	.compat_ioctl	= nvme_ioctl,
 	.open		= nvme_open,
 	.release	= nvme_release,
 	.getgeo		= nvme_getgeo,
-- 
2.14.1



From ckulkarnilinux at gmail.com  Thu Oct  5 12:48:22 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Thu,  5 Oct 2017 12:48:22 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
Message-ID: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>

This is the sixth version of the patch for adding SGL support
for nvme-pci driver.

Changes since v5:-

1. Remove the NVME_CTRL_SGL_NOT_SUPP enum constant and use
   simple comparison bits to determine whether controller supports
   SGL or not.

Changes since v4:-

1. Replace NVME_CTRL_SUPP_SGL macro with enum NVME_CTRL_SGL_NOT_SUPP.
2. Adjust patch for new enum.
3. Merge two patches into one.

Changes since v3:-

1. Modify SGL helper functions and add prefix nvme_pci_sgl.
2. Use average request physical segment(s) size to conditionally use SGLs.
   This approach is different from the original one, the original approach
   was based no the IO size. The new approach considers IO size and 
   number of physical segments present in the request to conditionally
   use SGLs.
3. Adjust the patch for Linux 4.14-rc3.

Changes since v2:-

1. Adjust the patch for latest nvme driver changes.
2. For initial submission use the default virt boundary values.
        
In future, work on the patch series for relaxing the virt boundary
for each transport.

Changes since the v1:-

1. Introduced nvme controller virt_boundary mask. For controllers that
   can handle arbitrarily sized bios (e.g advanced RDMA and PCI ctrls)
   we can allow the block layer to pass us gaps by setting the 
   queue.virt_boundary_mask according to each ctrl constraints.
2. Simplified code for setting up the SGLs.
3. Added SGL helper functions.
4. Modified commit log.
5. Use the newly introduced virt_boundary mask and clear its 
   value for SGL capable controllers in PCIe driver.

For reference v1 details:-

This adds SGL support for NVMe PCIe driver which is
reimplementation of the initial version provided by
Rajiv Shanmugam Madeswaran(smrajiv15 at gmail.com):-
"block: nvme-core: Scatter gather list support in the 
NVMe Block Driver".

In order to enable SGL mode for the driver, the user can set the 
sgl_threshold module parameter. This parameter can be used
in the two modes:-

1. Allow all IOs to use SGL mode. (set sgl_threshold to 1). 
2. Set up the IO size threshold to determine whether to use 
    SGLs or PRPs for each IO. (set sgl_threshold to 4096 to
    use SGL only for IOs which are >= 4096 in the size).

[PATCH V6] nvme-pci: add SGL support

-Regards,
Chaitanya


From ckulkarnilinux at gmail.com  Thu Oct  5 12:48:23 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Thu,  5 Oct 2017 12:48:23 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds SGL support for NVMe PCIe driver, based on an earlier patch
from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
refactors the original code and adds new module parameter sgl_threshold
to determine whether to use SGL or PRP for IOs.

The usage of SGLs is controlled by the sgl_threshold module parameter,
which allows to conditionally use SGLs if average request segment size
(avg_seg_size) is greater than sgl_threshold. In the original patch,
the decision of using SGLs was dependent only on the IO size,
with the new approach we consider not only IO size but also the
number of physical segments present in the IO.

We calculate avg_seg_size based on request payload bytes and number
of physical segments present in the request.

For e.g.:-

1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.

3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
avg_seg_size = 4K use sgl if avg_seg_size

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 222 ++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 197 insertions(+), 25 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cb73bc8..56db876 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -45,6 +45,8 @@
  */
 #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
 
+#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+
 static int use_threaded_interrupts;
 module_param(use_threaded_interrupts, int, 0);
 
@@ -57,6 +59,10 @@
 MODULE_PARM_DESC(max_host_mem_size_mb,
 	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
 
+static unsigned int sgl_threshold = SZ_32K;
+module_param(sgl_threshold, uint, 0644);
+MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this size");
+
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
 static const struct kernel_param_ops io_queue_depth_ops = {
 	.set = io_queue_depth_set,
@@ -178,6 +184,7 @@ struct nvme_queue {
 struct nvme_iod {
 	struct nvme_request req;
 	struct nvme_queue *nvmeq;
+	bool use_sgl;
 	int aborted;
 	int npages;		/* In the PRP list. 0 means small pool in use */
 	int nents;		/* Used in scatterlist */
@@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
 	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
 }
 
+/*
+ * Calculates the number of pages needed for the SGL segments. For example a 4k
+ * page can accommodate 256 SGL descriptors.
+ */
+static int nvme_npages_sgl(unsigned int num_seg)
+{
+	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
+}
+
 static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
-		unsigned int size, unsigned int nseg)
+		unsigned int size, unsigned int nseg, bool use_sgl)
 {
-	return sizeof(__le64 *) * nvme_npages(size, dev) +
-			sizeof(struct scatterlist) * nseg;
+	size_t alloc_size;
+
+	if (use_sgl)
+		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
+	else
+		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
+
+	return alloc_size + sizeof(struct scatterlist) * nseg;
 }
 
-static unsigned int nvme_cmd_size(struct nvme_dev *dev)
+static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
 {
 	return sizeof(struct nvme_iod) +
-		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
+		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
 }
 
 static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
@@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
 	nvmeq->sq_tail = tail;
 }
 
-static __le64 **iod_list(struct request *req)
+static void **iod_list(struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
+	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
 }
 
 static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
@@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 	unsigned int size = blk_rq_payload_bytes(rq);
 
 	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
-		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
+		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
+				iod->use_sgl);
+
+		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
 		if (!iod->sg)
 			return BLK_STS_RESOURCE;
 	} else {
@@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	const int last_prp = dev->ctrl.page_size / 8 - 1;
+	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
+	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
 	int i;
-	__le64 **list = iod_list(req);
-	dma_addr_t prp_dma = iod->first_dma;
 
 	if (iod->npages == 0)
-		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
+		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
+
 	for (i = 0; i < iod->npages; i++) {
-		__le64 *prp_list = list[i];
-		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
-		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
-		prp_dma = next_prp_dma;
+		void *addr = iod_list(req)[i];
+
+		if (iod->use_sgl) {
+			struct nvme_sgl_desc *sg_list = addr;
+
+			next_dma_addr =
+				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
+		} else {
+			__le64 *prp_list = addr;
+
+			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
+		}
+
+		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
+		dma_addr = next_dma_addr;
 	}
 
 	if (iod->sg != iod->inline_sg)
@@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
 	}
 }
 
-static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
+static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmnd)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
 	struct dma_pool *pool;
@@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	u32 page_size = dev->ctrl.page_size;
 	int offset = dma_addr & (page_size - 1);
 	__le64 *prp_list;
-	__le64 **list = iod_list(req);
+	void **list = iod_list(req);
 	dma_addr_t prp_dma;
 	int nprps, i;
 
+	iod->use_sgl = false;
+
 	length -= (page_size - offset);
 	if (length <= 0) {
 		iod->first_dma = 0;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	dma_len -= (page_size - offset);
@@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 
 	if (length <= page_size) {
 		iod->first_dma = dma_addr;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	nprps = DIV_ROUND_UP(length, page_size);
@@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 		dma_len = sg_dma_len(sg);
 	}
 
+done:
+	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
+	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
+
 	return BLK_STS_OK;
 
  bad_sgl:
@@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	return BLK_STS_IOERR;
 }
 
+static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
+		struct scatterlist *sg)
+{
+	sge->addr = cpu_to_le64(sg_dma_address(sg));
+	sge->length = cpu_to_le32(sg_dma_len(sg));
+	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr, int entries)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
+	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	sge->length = cpu_to_le32(PAGE_SIZE);
+	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
+}
+
+static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
+		struct nvme_rw_command *cmd)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	int length = blk_rq_payload_bytes(req);
+	struct dma_pool *pool;
+	struct nvme_sgl_desc *sg_list;
+	struct scatterlist *sg = iod->sg;
+	int entries = iod->nents, i = 0;
+	dma_addr_t sgl_dma;
+
+	iod->use_sgl = true;
+
+	cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as SGL */
+
+	if (length == sg_dma_len(sg)) {
+		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
+		return BLK_STS_OK;
+	}
+
+	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
+		pool = dev->prp_small_pool;
+		iod->npages = 0;
+	} else {
+		pool = dev->prp_page_pool;
+		iod->npages = 1;
+	}
+
+	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+	if (!sg_list) {
+		iod->npages = -1;
+		return BLK_STS_RESOURCE;
+	}
+
+	iod_list(req)[0] = sg_list;
+	iod->first_dma = sgl_dma;
+
+	if (entries <= SGES_PER_PAGE) {
+		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
+
+		for (i = 0; i < entries; i++) {
+			nvme_pci_sgl_set_data(&sg_list[i], sg);
+			length -= sg_dma_len(sg);
+			sg = sg_next(sg);
+		}
+
+		WARN_ON(length > 0);
+		return BLK_STS_OK;
+	}
+
+	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
+
+	do {
+		if (i == SGES_PER_PAGE) {
+			struct nvme_sgl_desc *old_sg_desc = sg_list;
+			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
+
+			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+			if (!sg_list)
+				return BLK_STS_RESOURCE;
+
+			i = 0;
+			iod_list(req)[iod->npages++] = sg_list;
+			sg_list[i++] = *link;
+
+			if (entries < SGES_PER_PAGE)
+				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
+			else
+				nvme_pci_sgl_set_seg(link, sgl_dma);
+		}
+
+		nvme_pci_sgl_set_data(&sg_list[i], sg);
+
+		length -= sg_dma_len(sg);
+		sg = sg_next(sg);
+		entries--;
+	} while (length > 0);
+
+	WARN_ON(entries > 0);
+	return BLK_STS_OK;
+}
+
+static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	unsigned int avg_seg_size;
+
+	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
+			blk_rq_nr_phys_segments(req));
+
+	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
+		return false;
+	if (!iod->nvmeq->qid)
+		return false;
+	if (!sgl_threshold || avg_seg_size < sgl_threshold)
+		return false;
+	return true;
+}
+
 static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 		struct nvme_command *cmnd)
 {
@@ -662,7 +828,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 				DMA_ATTR_NO_WARN))
 		goto out;
 
-	ret = nvme_setup_prps(dev, req);
+	if (nvme_pci_use_sgls(dev, req))
+		ret = nvme_setup_sgls(dev, req, &cmnd->rw);
+	else
+		ret = nvme_setup_prps(dev, req, &cmnd->rw);
+
 	if (ret != BLK_STS_OK)
 		goto out_unmap;
 
@@ -682,8 +852,6 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 			goto out_unmap;
 	}
 
-	cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
-	cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
 	if (blk_integrity_rq(req))
 		cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
 	return BLK_STS_OK;
@@ -1379,7 +1547,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
-		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
+		dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
 		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
 		dev->admin_tagset.driver_data = dev;
 
@@ -1906,7 +2074,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
 		dev->tagset.numa_node = dev_to_node(dev->dev);
 		dev->tagset.queue_depth =
 				min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
-		dev->tagset.cmd_size = nvme_cmd_size(dev);
+		dev->tagset.cmd_size = nvme_cmd_size(dev, false);
+		if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
+			dev->tagset.cmd_size = max(dev->tagset.cmd_size,
+					nvme_cmd_size(dev, true));
+		}
 		dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
 		dev->tagset.driver_data = dev;
 
-- 
1.8.3.1



From keith.busch at intel.com  Fri Oct  6 08:05:57 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri,  6 Oct 2017 09:05:57 -0600
Subject: [PATCH-stable] nvme: protect against simultaneous shutdown invocations
Message-ID: <20171006150557.17492-1-keith.busch@intel.com>

commit 77bf25ea70200cddf083f74b7f617e5f07fac8bd upstream

Back-ported to 4.4. The only difference is the file location of the
struct definition that's adding the mutex.

This fixes reported reported kernel panics in 4.4-stable from simultaneous
controller resets that never supposed to be allowed to happen.

Cc: <stable at vger.kernel.org> # 4.4
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/nvme.h | 2 ++
 drivers/nvme/host/pci.c  | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index b8a5a8e8f57d..88cf4f5025b0 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -14,6 +14,7 @@
 #ifndef _NVME_H
 #define _NVME_H
 
+#include <linux/mutex.h>
 #include <linux/nvme.h>
 #include <linux/pci.h>
 #include <linux/kref.h>
@@ -62,6 +63,7 @@ struct nvme_dev {
 	struct work_struct reset_work;
 	struct work_struct probe_work;
 	struct work_struct scan_work;
+	struct mutex shutdown_lock;
 	char name[12];
 	char serial[20];
 	char model[40];
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 4c673d45f1bd..a36072b926f8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2954,6 +2954,7 @@ static void nvme_dev_shutdown(struct nvme_dev *dev)
 
 	nvme_dev_list_remove(dev);
 
+	mutex_lock(&dev->shutdown_lock);
 	if (pci_is_enabled(to_pci_dev(dev->dev))) {
 		nvme_freeze_queues(dev);
 		csts = readl(&dev->bar->csts);
@@ -2972,6 +2973,7 @@ static void nvme_dev_shutdown(struct nvme_dev *dev)
 
 	for (i = dev->queue_count - 1; i >= 0; i--)
 		nvme_clear_queue(dev->queues[i]);
+	mutex_init(&dev->shutdown_lock);
 }
 
 static void nvme_dev_remove(struct nvme_dev *dev)
@@ -3328,6 +3330,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	INIT_LIST_HEAD(&dev->namespaces);
 	INIT_WORK(&dev->reset_work, nvme_reset_work);
+	mutex_init(&dev->shutdown_lock);
 	dev->dev = get_device(&pdev->dev);
 	pci_set_drvdata(pdev, dev);
 
-- 
2.13.6



From keith.busch at intel.com  Fri Oct  6 08:09:13 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 6 Oct 2017 09:09:13 -0600
Subject: [PATCH-stable] nvme: protect against simultaneous shutdown
 invocations
In-Reply-To: <20171006150557.17492-1-keith.busch@intel.com>
References: <20171006150557.17492-1-keith.busch@intel.com>
Message-ID: <20171006150912.GA17569@localhost.localdomain>

On Fri, Oct 06, 2017 at 09:05:57AM -0600, Keith Busch wrote:
> commit 77bf25ea70200cddf083f74b7f617e5f07fac8bd upstream
> 
> Back-ported to 4.4. The only difference is the file location of the
> struct definition that's adding the mutex.

Err, sorry, please disregard this one. There is a mistake in the back
port that will break. Will send out the correct version.


From keith.busch at intel.com  Fri Oct  6 08:14:55 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri,  6 Oct 2017 09:14:55 -0600
Subject: [PATCH-stable] nvme: protect against simultaneous shutdown invocations
Message-ID: <20171006151455.18816-1-keith.busch@intel.com>

commit 77bf25ea70200cddf083f74b7f617e5f07fac8bd upstream

Back-ported to 4.4. The difference is the file location of the struct
definition that's adding the mutex.

This fixes reported kernel panics in 4.4-stable from simultaneous
controller resets that was never supposed to be allowed to happen.

Cc: <stable at vger.kernel.org> # 4.4
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/nvme.h | 2 ++
 drivers/nvme/host/pci.c  | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index b8a5a8e8f57d..88cf4f5025b0 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -14,6 +14,7 @@
 #ifndef _NVME_H
 #define _NVME_H
 
+#include <linux/mutex.h>
 #include <linux/nvme.h>
 #include <linux/pci.h>
 #include <linux/kref.h>
@@ -62,6 +63,7 @@ struct nvme_dev {
 	struct work_struct reset_work;
 	struct work_struct probe_work;
 	struct work_struct scan_work;
+	struct mutex shutdown_lock;
 	char name[12];
 	char serial[20];
 	char model[40];
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 4c673d45f1bd..a36072b926f8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2954,6 +2954,7 @@ static void nvme_dev_shutdown(struct nvme_dev *dev)
 
 	nvme_dev_list_remove(dev);
 
+	mutex_lock(&dev->shutdown_lock);
 	if (pci_is_enabled(to_pci_dev(dev->dev))) {
 		nvme_freeze_queues(dev);
 		csts = readl(&dev->bar->csts);
@@ -2972,6 +2973,7 @@ static void nvme_dev_shutdown(struct nvme_dev *dev)
 
 	for (i = dev->queue_count - 1; i >= 0; i--)
 		nvme_clear_queue(dev->queues[i]);
+	mutex_unlock(&dev->shutdown_lock);
 }
 
 static void nvme_dev_remove(struct nvme_dev *dev)
@@ -3328,6 +3330,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	INIT_LIST_HEAD(&dev->namespaces);
 	INIT_WORK(&dev->reset_work, nvme_reset_work);
+	mutex_init(&dev->shutdown_lock);
 	dev->dev = get_device(&pdev->dev);
 	pci_set_drvdata(pdev, dev);
 
-- 
2.13.6



From yogi43222 at gmail.com  Fri Oct  6 14:49:06 2017
From: yogi43222 at gmail.com (Yogi Wannabe)
Date: Fri, 6 Oct 2017 14:49:06 -0700
Subject: Missing interrupts in kernel 4.5.0-21.cavium.el7.aarch64
Message-ID: <CAGSHeY1uhgMT5oQBt8fxEzO1k=s9q9eBaFGm+oVt9a31_Ga9aA@mail.gmail.com>

Hi,

Our NVMe controller works fine with Kernel 3.9.xx. On kernel
4.5.0-21.cavium.el7.aarch64 we see some issues with interrupts.

Upon nvme driver load it, sends an Identify cmd.. The controller
processes the cmd and sends a completion interrupt. The driver does
not see this interrupt nor is it visible in /proc/interrupts. After
admin timeout duration, driver disables controller and calls
free_irq(). As soon as free_irq() is called, interrupt reaches driver
and nvme_irq() is called. The interrupt vector is usually 17.

Are there any known issues that could be causing this weird behavior?

TIA
Yogi


From jsmart2021 at gmail.com  Sat Oct  7 08:53:37 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 7 Oct 2017 08:53:37 -0700
Subject: [PATCH v2 2/7] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20171005075730.GA5063@infradead.org>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-3-jsmart2021@gmail.com>
 <20171005075730.GA5063@infradead.org>
Message-ID: <95dd2176-c5bb-386f-6a7a-4705576713fd@gmail.com>

On 10/5/2017 12:57 AM, Christoph Hellwig wrote:
>>   	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
>> -	WARN_ON_ONCE(!changed);
>>   
>>   	ctrl->ctrl.nr_reconnects = 0;
>>   
>> -	nvme_start_ctrl(&ctrl->ctrl);
>> +	if (changed)
>> +		nvme_start_ctrl(&ctrl->ctrl);
> 
> It's just cosmetic, but can you folow the RDMA pattern here:
> 
> 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
> 	if (!changed) {
> 		/* state change failure is ok if we're in DELETING state */
> 		WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
> 		return;
> 	}
> 
> else the patch looks fine:
> 
> Reviewed-by: Christoph Hellwig <hch at lst.de>
> 

Actually, it's not just cosmetic. I don't want the warning to be 
displayed. With FC's notification on device loss, in testing, we've seen 
a device loss occur as this state transition occurs. I doubt that's 
something rdma would ever see. With the device connectivity loss 
notification, the state moves from NEW to RESETTING, then RECONNECTING. 
I prefer to clear the reconnect counter (as we just went through a 
connect) to ensure that the full reconnect attempts will occur.

As such, I'd prefer to leave it the same.

-- james



From jsmart2021 at gmail.com  Sat Oct  7 09:08:21 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 7 Oct 2017 09:08:21 -0700
Subject: [PATCH v2 4/7] nvme_fc: add dev_loss_tmo to controller
In-Reply-To: <20171005080040.GB5063@infradead.org>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-5-jsmart2021@gmail.com>
 <20171005080040.GB5063@infradead.org>
Message-ID: <9c9b8f0d-cc31-91ec-1be8-b5fd616dd0c5@gmail.com>

On 10/5/2017 1:00 AM, Christoph Hellwig wrote:
> Can you please add the dev_loss_tmo to the generic nvme_ctrl and build
> the infrastructure in common code?
> 
> Then as a next step we can look into inferring it from the remote port
> for FC.
> 

Hmm - this raise flags in my mind. I'm not sure how a dev_loss_tmo 
applies to any other transport other than FC.

I am inclined to rework this a bit though as I think the mashing of the 
dev_loss_tmo value into the controller's ctrl_loss_tmo has caused 
confusion. The dev_loss_tmo should stay in the transport and be specific 
to the remoteport.  Mashing them also caused us to loose the initial 
creation ctrl_loss_tmo value if there had been one set and the dev_loss 
value dynamically changed.  I'll rework it so that: dev_loss_tmo is an 
FC internal timer that acts on the remoteport.  Upon device loss, the 
remote ports dev_loss timer will be started, and any controllers on the 
remote port will have their associations torn down and their 
ctrl_loss_tmo (aka max_reconnects and reconnect_delay) started. If a 
controllers max_reconnects is hit it will be deleted. If the rport's 
dev_loss_tmo is hit, it will delete any suspended controller.

-- james




From hch at infradead.org  Sat Oct  7 23:24:08 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Sat, 7 Oct 2017 23:24:08 -0700
Subject: Missing interrupts in kernel 4.5.0-21.cavium.el7.aarch64
In-Reply-To: <CAGSHeY1uhgMT5oQBt8fxEzO1k=s9q9eBaFGm+oVt9a31_Ga9aA@mail.gmail.com>
References: <CAGSHeY1uhgMT5oQBt8fxEzO1k=s9q9eBaFGm+oVt9a31_Ga9aA@mail.gmail.com>
Message-ID: <20171008062407.GA20749@infradead.org>

On Fri, Oct 06, 2017 at 02:49:06PM -0700, Yogi Wannabe wrote:
> Are there any known issues that could be causing this weird behavior?

Probablt broken interrupt handling code in your patched non-mainline
code.


From maxg at mellanox.com  Sun Oct  8 02:16:10 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sun, 8 Oct 2017 12:16:10 +0300
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>



On 10/5/2017 10:48 PM, Chaitanya Kulkarni wrote:
> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> 
> This adds SGL support for NVMe PCIe driver, based on an earlier patch
> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
> refactors the original code and adds new module parameter sgl_threshold
> to determine whether to use SGL or PRP for IOs.
> 
> The usage of SGLs is controlled by the sgl_threshold module parameter,
> which allows to conditionally use SGLs if average request segment size
> (avg_seg_size) is greater than sgl_threshold. In the original patch,
> the decision of using SGLs was dependent only on the IO size,
> with the new approach we consider not only IO size but also the
> number of physical segments present in the IO.
> 
> We calculate avg_seg_size based on request payload bytes and number
> of physical segments present in the request.

Can you add some performance numbers here to show the benefit ?
and also please recomend the user when it's better to use sgls vs prps.

> 
> For e.g.:-
> 
> 1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
> avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.
> 
> 2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
> avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.
> 
> 3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
> avg_seg_size = 4K use sgl if avg_seg_size

typo :)

> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>   drivers/nvme/host/pci.c | 222 ++++++++++++++++++++++++++++++++++++++++++------
>   1 file changed, 197 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index cb73bc8..56db876 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -45,6 +45,8 @@
>    */
>   #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
>   
> +#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))

should we use PAGE_SIZE or ctrl->page_size ?

> +
>   static int use_threaded_interrupts;
>   module_param(use_threaded_interrupts, int, 0);
>   
> @@ -57,6 +59,10 @@
>   MODULE_PARM_DESC(max_host_mem_size_mb,
>   	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
>   
> +static unsigned int sgl_threshold = SZ_32K;
> +module_param(sgl_threshold, uint, 0644);
> +MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this size");
> +

please mention that 0 value disable sgl usage.

>   static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
>   static const struct kernel_param_ops io_queue_depth_ops = {
>   	.set = io_queue_depth_set,
> @@ -178,6 +184,7 @@ struct nvme_queue {
>   struct nvme_iod {
>   	struct nvme_request req;
>   	struct nvme_queue *nvmeq;
> +	bool use_sgl;
>   	int aborted;
>   	int npages;		/* In the PRP list. 0 means small pool in use */
>   	int nents;		/* Used in scatterlist */
> @@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
>   	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
>   }
>   
> +/*
> + * Calculates the number of pages needed for the SGL segments. For example a 4k
> + * page can accommodate 256 SGL descriptors.
> + */
> +static int nvme_npages_sgl(unsigned int num_seg)
> +{
> +	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
> +}
> +
>   static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
> -		unsigned int size, unsigned int nseg)
> +		unsigned int size, unsigned int nseg, bool use_sgl)
>   {
> -	return sizeof(__le64 *) * nvme_npages(size, dev) +
> -			sizeof(struct scatterlist) * nseg;
> +	size_t alloc_size;
> +
> +	if (use_sgl)
> +		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
> +	else
> +		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
> +
> +	return alloc_size + sizeof(struct scatterlist) * nseg;
>   }
>   
> -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>   {
>   	return sizeof(struct nvme_iod) +
> -		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
> +		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
>   }
>   
>   static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
>   	nvmeq->sq_tail = tail;
>   }
>   
> -static __le64 **iod_list(struct request *req)
> +static void **iod_list(struct request *req)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
> +	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>   }
>   
>   static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>   	unsigned int size = blk_rq_payload_bytes(rq);
>   
>   	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
> -		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
> +		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
> +				iod->use_sgl);
> +
> +		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>   		if (!iod->sg)
>   			return BLK_STS_RESOURCE;
>   	} else {
> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>   static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	const int last_prp = dev->ctrl.page_size / 8 - 1;
> +	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
> +	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>   	int i;
> -	__le64 **list = iod_list(req);
> -	dma_addr_t prp_dma = iod->first_dma;
>   
>   	if (iod->npages == 0)
> -		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
> +		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
> +
>   	for (i = 0; i < iod->npages; i++) {
> -		__le64 *prp_list = list[i];
> -		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
> -		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
> -		prp_dma = next_prp_dma;
> +		void *addr = iod_list(req)[i];
> +
> +		if (iod->use_sgl) {
> +			struct nvme_sgl_desc *sg_list = addr;
> +
> +			next_dma_addr =
> +				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
> +		} else {
> +			__le64 *prp_list = addr;
> +
> +			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
> +		}
> +
> +		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
> +		dma_addr = next_dma_addr;
>   	}
>   
>   	if (iod->sg != iod->inline_sg)
> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
>   	}
>   }
>   
> -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmnd)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>   	struct dma_pool *pool;
> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   	u32 page_size = dev->ctrl.page_size;
>   	int offset = dma_addr & (page_size - 1);
>   	__le64 *prp_list;
> -	__le64 **list = iod_list(req);
> +	void **list = iod_list(req);
>   	dma_addr_t prp_dma;
>   	int nprps, i;
>   
> +	iod->use_sgl = false;
> +
>   	length -= (page_size - offset);
>   	if (length <= 0) {
>   		iod->first_dma = 0;
> -		return BLK_STS_OK;
> +		goto done;
>   	}
>   
>   	dma_len -= (page_size - offset);
> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   
>   	if (length <= page_size) {
>   		iod->first_dma = dma_addr;
> -		return BLK_STS_OK;
> +		goto done;
>   	}
>   
>   	nprps = DIV_ROUND_UP(length, page_size);
> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   		dma_len = sg_dma_len(sg);
>   	}
>   
> +done:
> +	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
> +	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
> +
>   	return BLK_STS_OK;
>   
>    bad_sgl:
> @@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   	return BLK_STS_IOERR;
>   }
>   
> +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
> +		struct scatterlist *sg)
> +{
> +	sge->addr = cpu_to_le64(sg_dma_address(sg));
> +	sge->length = cpu_to_le32(sg_dma_len(sg));
> +	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr, int entries)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
> +	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(PAGE_SIZE);
> +	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
> +}
> +
> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmd)
> +{
> +	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> +	int length = blk_rq_payload_bytes(req);
> +	struct dma_pool *pool;
> +	struct nvme_sgl_desc *sg_list;
> +	struct scatterlist *sg = iod->sg;
> +	int entries = iod->nents, i = 0;
> +	dma_addr_t sgl_dma;
> +
> +	iod->use_sgl = true;
> +
> +	cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as SGL */
> +
> +	if (length == sg_dma_len(sg)) {
> +		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
> +		return BLK_STS_OK;
> +	}
> +
> +	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {

where does 256 comes from ? can we use macro definition here ?

> +		pool = dev->prp_small_pool;
> +		iod->npages = 0;
> +	} else {
> +		pool = dev->prp_page_pool;
> +		iod->npages = 1;
> +	}
> +
> +	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
> +	if (!sg_list) {
> +		iod->npages = -1;
> +		return BLK_STS_RESOURCE;
> +	}
> +
> +	iod_list(req)[0] = sg_list;
> +	iod->first_dma = sgl_dma;
> +
> +	if (entries <= SGES_PER_PAGE) {
> +		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
> +
> +		for (i = 0; i < entries; i++) {
> +			nvme_pci_sgl_set_data(&sg_list[i], sg);
> +			length -= sg_dma_len(sg);
> +			sg = sg_next(sg);
> +		}
> +
> +		WARN_ON(length > 0);
> +		return BLK_STS_OK;
> +	}
> +
> +	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
> +
> +	do {
> +		if (i == SGES_PER_PAGE) {
> +			struct nvme_sgl_desc *old_sg_desc = sg_list;
> +			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
> +
> +			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
> +			if (!sg_list)
> +				return BLK_STS_RESOURCE;
> +
> +			i = 0;
> +			iod_list(req)[iod->npages++] = sg_list;
> +			sg_list[i++] = *link;
> +
> +			if (entries < SGES_PER_PAGE)
> +				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
> +			else
> +				nvme_pci_sgl_set_seg(link, sgl_dma);
> +		}
> +
> +		nvme_pci_sgl_set_data(&sg_list[i], sg);
> +
> +		length -= sg_dma_len(sg);
> +		sg = sg_next(sg);
> +		entries--;
> +	} while (length > 0);
> +
> +	WARN_ON(entries > 0);
> +	return BLK_STS_OK;
> +}
> +
> +static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
> +{
> +	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> +	unsigned int avg_seg_size;
> +
> +	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
> +			blk_rq_nr_phys_segments(req));
> +
> +	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))

Can you create macros for the spec definitions ?
In case of Dword alignment and granularity, where do we check that 
requirement ?

> +		return false;
> +	if (!iod->nvmeq->qid)
> +		return false;
> +	if (!sgl_threshold || avg_seg_size < sgl_threshold)
> +		return false;
> +	return true;
> +}
> +
>   static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
>   		struct nvme_command *cmnd)
>   {
> @@ -662,7 +828,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
>   				DMA_ATTR_NO_WARN))
>   		goto out;
>   
> -	ret = nvme_setup_prps(dev, req);
> +	if (nvme_pci_use_sgls(dev, req))
> +		ret = nvme_setup_sgls(dev, req, &cmnd->rw);
> +	else
> +		ret = nvme_setup_prps(dev, req, &cmnd->rw);
> +
>   	if (ret != BLK_STS_OK)
>   		goto out_unmap;
>   
> @@ -682,8 +852,6 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
>   			goto out_unmap;
>   	}
>   
> -	cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
> -	cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
>   	if (blk_integrity_rq(req))
>   		cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
>   	return BLK_STS_OK;
> @@ -1379,7 +1547,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
>   		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
>   		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
>   		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
> -		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
> +		dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
>   		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
>   		dev->admin_tagset.driver_data = dev;
>   
> @@ -1906,7 +2074,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
>   		dev->tagset.numa_node = dev_to_node(dev->dev);
>   		dev->tagset.queue_depth =
>   				min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
> -		dev->tagset.cmd_size = nvme_cmd_size(dev);
> +		dev->tagset.cmd_size = nvme_cmd_size(dev, false);
> +		if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
> +			dev->tagset.cmd_size = max(dev->tagset.cmd_size,
> +					nvme_cmd_size(dev, true));
> +		}
>   		dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
>   		dev->tagset.driver_data = dev;
>   
> 


From maxg at mellanox.com  Sun Oct  8 02:32:15 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sun, 8 Oct 2017 12:32:15 +0300
Subject: [PATCH] nvme: simplify compat_ioctl handling
In-Reply-To: <20171005164646.17473-1-hch@lst.de>
References: <20171005164646.17473-1-hch@lst.de>
Message-ID: <3b12ebd1-7282-a807-25e8-7cabde16486a@mellanox.com>



On 10/5/2017 7:46 PM, Christoph Hellwig wrote:
> We can just use our normal ioctl handler for the compat case and remove
> the boilerplate code for it.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>   drivers/nvme/host/core.c | 12 +-----------
>   1 file changed, 1 insertion(+), 11 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 26c8913435b2..573cc3b59bfa 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c

Looks good,
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From maxg at mellanox.com  Sun Oct  8 02:57:15 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sun, 8 Oct 2017 12:57:15 +0300
Subject: [Suspected-Phishing]Re: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
 <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
Message-ID: <0c37ee0a-ea61-094b-eaa9-198f5c01722a@mellanox.com>



On 10/8/2017 12:16 PM, Max Gurtovoy wrote:
> 
> 
> On 10/5/2017 10:48 PM, Chaitanya Kulkarni wrote:
>> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>>
>> This adds SGL support for NVMe PCIe driver, based on an earlier patch
>> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
>> refactors the original code and adds new module parameter sgl_threshold
>> to determine whether to use SGL or PRP for IOs.
>>
>> The usage of SGLs is controlled by the sgl_threshold module parameter,
>> which allows to conditionally use SGLs if average request segment size
>> (avg_seg_size) is greater than sgl_threshold. In the original patch,
>> the decision of using SGLs was dependent only on the IO size,
>> with the new approach we consider not only IO size but also the
>> number of physical segments present in the IO.
>>
>> We calculate avg_seg_size based on request payload bytes and number
>> of physical segments present in the request.
> 
> Can you add some performance numbers here to show the benefit ?
> and also please recomend the user when it's better to use sgls vs prps.
> 
>>
>> For e.g.:-
>>
>> 1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
>> avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.
>>
>> 2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
>> avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.
>>
>> 3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
>> avg_seg_size = 4K use sgl if avg_seg_size
> 
> typo :)
> 
>>
>> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>> ---
>> ? drivers/nvme/host/pci.c | 222 
>> ++++++++++++++++++++++++++++++++++++++++++------
>> ? 1 file changed, 197 insertions(+), 25 deletions(-)
>>
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index cb73bc8..56db876 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -45,6 +45,8 @@
>> ?? */
>> ? #define NVME_AQ_BLKMQ_DEPTH??? (NVME_AQ_DEPTH - NVME_NR_AERS)
>> +#define SGES_PER_PAGE????????? (PAGE_SIZE / sizeof(struct 
>> nvme_sgl_desc))
> 
> should we use PAGE_SIZE or ctrl->page_size ?
> 
>> +
>> ? static int use_threaded_interrupts;
>> ? module_param(use_threaded_interrupts, int, 0);
>> @@ -57,6 +59,10 @@
>> ? MODULE_PARM_DESC(max_host_mem_size_mb,
>> ????? "Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
>> +static unsigned int sgl_threshold = SZ_32K;
>> +module_param(sgl_threshold, uint, 0644);
>> +MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to 
>> this size");
>> +
> 
> please mention that 0 value disable sgl usage.
> 
>> ? static int io_queue_depth_set(const char *val, const struct 
>> kernel_param *kp);
>> ? static const struct kernel_param_ops io_queue_depth_ops = {
>> ????? .set = io_queue_depth_set,
>> @@ -178,6 +184,7 @@ struct nvme_queue {
>> ? struct nvme_iod {
>> ????? struct nvme_request req;
>> ????? struct nvme_queue *nvmeq;
>> +??? bool use_sgl;
>> ????? int aborted;
>> ????? int npages;??????? /* In the PRP list. 0 means small pool in use */
>> ????? int nents;??????? /* Used in scatterlist */
>> @@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct 
>> nvme_dev *dev)
>> ????? return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
>> ? }
>> +/*
>> + * Calculates the number of pages needed for the SGL segments. For 
>> example a 4k
>> + * page can accommodate 256 SGL descriptors.
>> + */
>> +static int nvme_npages_sgl(unsigned int num_seg)
>> +{
>> +??? return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), 
>> PAGE_SIZE);
>> +}
>> +
>> ? static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
>> -??????? unsigned int size, unsigned int nseg)
>> +??????? unsigned int size, unsigned int nseg, bool use_sgl)
>> ? {
>> -??? return sizeof(__le64 *) * nvme_npages(size, dev) +
>> -??????????? sizeof(struct scatterlist) * nseg;
>> +??? size_t alloc_size;
>> +
>> +??? if (use_sgl)
>> +??????? alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
>> +??? else
>> +??????? alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
>> +
>> +??? return alloc_size + sizeof(struct scatterlist) * nseg;
>> ? }
>> -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
>> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>> ? {
>> ????? return sizeof(struct nvme_iod) +
>> -??????? nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
>> +??????? nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, 
>> use_sgl);
>> ? }
>> ? static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
>> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue 
>> *nvmeq,
>> ????? nvmeq->sq_tail = tail;
>> ? }
>> -static __le64 **iod_list(struct request *req)
>> +static void **iod_list(struct request *req)
>> ? {
>> ????? struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -??? return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
>> +??? return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>> ? }
>> ? static blk_status_t nvme_init_iod(struct request *rq, struct 
>> nvme_dev *dev)
>> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request 
>> *rq, struct nvme_dev *dev)
>> ????? unsigned int size = blk_rq_payload_bytes(rq);
>> ????? if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
>> -??????? iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), 
>> GFP_ATOMIC);
>> +??????? size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
>> +??????????????? iod->use_sgl);
>> +
>> +??????? iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>> ????????? if (!iod->sg)
>> ????????????? return BLK_STS_RESOURCE;
>> ????? } else {
>> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request 
>> *rq, struct nvme_dev *dev)
>> ? static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>> ? {
>> ????? struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -??? const int last_prp = dev->ctrl.page_size / 8 - 1;
>> +??? const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
>> +??? dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>> ????? int i;
>> -??? __le64 **list = iod_list(req);
>> -??? dma_addr_t prp_dma = iod->first_dma;
>> ????? if (iod->npages == 0)
>> -??????? dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
>> +??????? dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
>> +
>> ????? for (i = 0; i < iod->npages; i++) {
>> -??????? __le64 *prp_list = list[i];
>> -??????? dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
>> -??????? dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
>> -??????? prp_dma = next_prp_dma;
>> +??????? void *addr = iod_list(req)[i];
>> +
>> +??????? if (iod->use_sgl) {
>> +??????????? struct nvme_sgl_desc *sg_list = addr;
>> +
>> +??????????? next_dma_addr =
>> +??????????????? le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
>> +??????? } else {
>> +??????????? __le64 *prp_list = addr;
>> +
>> +??????????? next_dma_addr = le64_to_cpu(prp_list[last_prp]);
>> +??????? }
>> +
>> +??????? dma_pool_free(dev->prp_page_pool, addr, dma_addr);
>> +??????? dma_addr = next_dma_addr;
>> ????? }
>> ????? if (iod->sg != iod->inline_sg)
>> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist 
>> *sgl, int nents)
>> ????? }
>> ? }
>> -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct 
>> request *req)
>> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct 
>> request *req,
>> +??????? struct nvme_rw_command *cmnd)
>> ? {
>> ????? struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> ????? struct dma_pool *pool;
>> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct 
>> nvme_dev *dev, struct request *req)
>> ????? u32 page_size = dev->ctrl.page_size;
>> ????? int offset = dma_addr & (page_size - 1);
>> ????? __le64 *prp_list;
>> -??? __le64 **list = iod_list(req);
>> +??? void **list = iod_list(req);
>> ????? dma_addr_t prp_dma;
>> ????? int nprps, i;
>> +??? iod->use_sgl = false;
>> +
>> ????? length -= (page_size - offset);
>> ????? if (length <= 0) {
>> ????????? iod->first_dma = 0;
>> -??????? return BLK_STS_OK;
>> +??????? goto done;
>> ????? }
>> ????? dma_len -= (page_size - offset);
>> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct 
>> nvme_dev *dev, struct request *req)
>> ????? if (length <= page_size) {
>> ????????? iod->first_dma = dma_addr;
>> -??????? return BLK_STS_OK;
>> +??????? goto done;
>> ????? }
>> ????? nprps = DIV_ROUND_UP(length, page_size);
>> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct 
>> nvme_dev *dev, struct request *req)
>> ????????? dma_len = sg_dma_len(sg);
>> ????? }
>> +done:
>> +??? cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
>> +??? cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
>> +
>> ????? return BLK_STS_OK;
>> ?? bad_sgl:
>> @@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct 
>> nvme_dev *dev, struct request *req)
>> ????? return BLK_STS_IOERR;
>> ? }
>> +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
>> +??????? struct scatterlist *sg)
>> +{
>> +??? sge->addr = cpu_to_le64(sg_dma_address(sg));
>> +??? sge->length = cpu_to_le32(sg_dma_len(sg));
>> +??? sge->type = NVME_SGL_FMT_DATA_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
>> +??????? dma_addr_t dma_addr, int entries)
>> +{
>> +??? sge->addr = cpu_to_le64(dma_addr);
>> +??? sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
>> +??? sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
>> +??????? dma_addr_t dma_addr)
>> +{
>> +??? sge->addr = cpu_to_le64(dma_addr);
>> +??? sge->length = cpu_to_le32(PAGE_SIZE);
>> +??? sge->type = NVME_SGL_FMT_SEG_DESC << 4;
>> +}
>> +
>> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct 
>> request *req,
>> +??????? struct nvme_rw_command *cmd)
>> +{
>> +??? struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> +??? int length = blk_rq_payload_bytes(req);
>> +??? struct dma_pool *pool;
>> +??? struct nvme_sgl_desc *sg_list;
>> +??? struct scatterlist *sg = iod->sg;
>> +??? int entries = iod->nents, i = 0;
>> +??? dma_addr_t sgl_dma;
>> +
>> +??? iod->use_sgl = true;
>> +
>> +??? cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type 
>> as SGL */
>> +
>> +??? if (length == sg_dma_len(sg)) {
>> +??????? nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
>> +??????? return BLK_STS_OK;
>> +??? }
>> +
>> +??? if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
> 
> where does 256 comes from ? can we use macro definition here ?
> 
>> +??????? pool = dev->prp_small_pool;
>> +??????? iod->npages = 0;
>> +??? } else {
>> +??????? pool = dev->prp_page_pool;
>> +??????? iod->npages = 1;
>> +??? }
>> +
>> +??? sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
>> +??? if (!sg_list) {
>> +??????? iod->npages = -1;
>> +??????? return BLK_STS_RESOURCE;
>> +??? }
>> +
>> +??? iod_list(req)[0] = sg_list;
>> +??? iod->first_dma = sgl_dma;
>> +
>> +??? if (entries <= SGES_PER_PAGE) {
>> +??????? nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
>> +
>> +??????? for (i = 0; i < entries; i++) {
>> +??????????? nvme_pci_sgl_set_data(&sg_list[i], sg);
>> +??????????? length -= sg_dma_len(sg);
>> +??????????? sg = sg_next(sg);
>> +??????? }
>> +
>> +??????? WARN_ON(length > 0);
>> +??????? return BLK_STS_OK;
>> +??? }
>> +
>> +??? nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
>> +
>> +??? do {
>> +??????? if (i == SGES_PER_PAGE) {
>> +??????????? struct nvme_sgl_desc *old_sg_desc = sg_list;
>> +??????????? struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
>> +
>> +??????????? sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
>> +??????????? if (!sg_list)
>> +??????????????? return BLK_STS_RESOURCE;
>> +
>> +??????????? i = 0;
>> +??????????? iod_list(req)[iod->npages++] = sg_list;
>> +??????????? sg_list[i++] = *link;
>> +
>> +??????????? if (entries < SGES_PER_PAGE)
>> +??????????????? nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
>> +??????????? else
>> +??????????????? nvme_pci_sgl_set_seg(link, sgl_dma);
>> +??????? }
>> +
>> +??????? nvme_pci_sgl_set_data(&sg_list[i], sg);
>> +
>> +??????? length -= sg_dma_len(sg);
>> +??????? sg = sg_next(sg);
>> +??????? entries--;
>> +??? } while (length > 0);
>> +
>> +??? WARN_ON(entries > 0);
>> +??? return BLK_STS_OK;
>> +}
>> +
>> +static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct 
>> request *req)
>> +{
>> +??? struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> +??? unsigned int avg_seg_size;
>> +
>> +??? avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
>> +??????????? blk_rq_nr_phys_segments(req));
>> +
>> +??? if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
> 
> Can you create macros for the spec definitions ?

Answering to myself - I saw that you discussed this already so I guess 
you can stay as is.

> In case of Dword alignment and granularity, where do we check that 
> requirement ?
> 
>> +??????? return false;
>> +??? if (!iod->nvmeq->qid)
>> +??????? return false;
>> +??? if (!sgl_threshold || avg_seg_size < sgl_threshold)
>> +??????? return false;
>> +??? return true;
>> +}
>> +
>> ? static blk_status_t nvme_map_data(struct nvme_dev *dev, struct 
>> request *req,
>> ????????? struct nvme_command *cmnd)
>> ? {
>> @@ -662,7 +828,11 @@ static blk_status_t nvme_map_data(struct nvme_dev 
>> *dev, struct request *req,
>> ????????????????? DMA_ATTR_NO_WARN))
>> ????????? goto out;
>> -??? ret = nvme_setup_prps(dev, req);
>> +??? if (nvme_pci_use_sgls(dev, req))
>> +??????? ret = nvme_setup_sgls(dev, req, &cmnd->rw);
>> +??? else
>> +??????? ret = nvme_setup_prps(dev, req, &cmnd->rw);
>> +
>> ????? if (ret != BLK_STS_OK)
>> ????????? goto out_unmap;
>> @@ -682,8 +852,6 @@ static blk_status_t nvme_map_data(struct nvme_dev 
>> *dev, struct request *req,
>> ????????????? goto out_unmap;
>> ????? }
>> -??? cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
>> -??? cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
>> ????? if (blk_integrity_rq(req))
>> ????????? cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
>> ????? return BLK_STS_OK;
>> @@ -1379,7 +1547,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev 
>> *dev)
>> ????????? dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
>> ????????? dev->admin_tagset.timeout = ADMIN_TIMEOUT;
>> ????????? dev->admin_tagset.numa_node = dev_to_node(dev->dev);
>> -??????? dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
>> +??????? dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
>> ????????? dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
>> ????????? dev->admin_tagset.driver_data = dev;
>> @@ -1906,7 +2074,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
>> ????????? dev->tagset.numa_node = dev_to_node(dev->dev);
>> ????????? dev->tagset.queue_depth =
>> ????????????????? min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
>> -??????? dev->tagset.cmd_size = nvme_cmd_size(dev);
>> +??????? dev->tagset.cmd_size = nvme_cmd_size(dev, false);
>> +??????? if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
>> +??????????? dev->tagset.cmd_size = max(dev->tagset.cmd_size,
>> +??????????????????? nvme_cmd_size(dev, true));
>> +??????? }
>> ????????? dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
>> ????????? dev->tagset.driver_data = dev;
>>
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7Cb2e5265cf8e94d7008f808d50e2d6e8a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636430510665543336&sdata=1hsl4jawK03SzkUe3AFEGgF3UAeFaAaAWbn2VtTyUEQ%3D&reserved=0 
> 


From keith.busch at intel.com  Mon Oct  9 07:49:45 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 9 Oct 2017 08:49:45 -0600
Subject: Missing interrupts in kernel 4.5.0-21.cavium.el7.aarch64
In-Reply-To: <CAGSHeY1uhgMT5oQBt8fxEzO1k=s9q9eBaFGm+oVt9a31_Ga9aA@mail.gmail.com>
References: <CAGSHeY1uhgMT5oQBt8fxEzO1k=s9q9eBaFGm+oVt9a31_Ga9aA@mail.gmail.com>
Message-ID: <20171009144945.GB19329@localhost.localdomain>

On Fri, Oct 06, 2017 at 02:49:06PM -0700, Yogi Wannabe wrote:
> Our NVMe controller works fine with Kernel 3.9.xx. On kernel
> 4.5.0-21.cavium.el7.aarch64 we see some issues with interrupts.
> 
> Upon nvme driver load it, sends an Identify cmd.. The controller
> processes the cmd and sends a completion interrupt. The driver does
> not see this interrupt nor is it visible in /proc/interrupts. After
> admin timeout duration, driver disables controller and calls
> free_irq(). As soon as free_irq() is called, interrupt reaches driver
> and nvme_irq() is called. The interrupt vector is usually 17.
> 
> Are there any known issues that could be causing this weird behavior?

4.5 removed the completion queue polling, relying exclusively on
interrupts to find these sorts of problems.

After that, lots of people reported failures with device initialization.

I don't believe the root caused was found since no issues were reported
when we enabled MSI/MSIx for early init in 4.6.

For more details, see:

  http://lists.infradead.org/pipermail/linux-nvme/2016-May/004653.html

This unfortauntely was not picked up by the 4.5-stable.


From keith.busch at intel.com  Mon Oct  9 08:12:11 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 9 Oct 2017 09:12:11 -0600
Subject: [PATCH] nvme: simplify compat_ioctl handling
In-Reply-To: <20171005164646.17473-1-hch@lst.de>
References: <20171005164646.17473-1-hch@lst.de>
Message-ID: <20171009151210.GC19329@localhost.localdomain>

On Thu, Oct 05, 2017 at 06:46:46PM +0200, Christoph Hellwig wrote:
> We can just use our normal ioctl handler for the compat case and remove
> the boilerplate code for it.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>

Looks good. The compact ioctl was left-over from SG_IO handling.

Reviewed-by: Keith Busch <keith.busch at intel.com>


From keith.busch at intel.com  Mon Oct  9 11:03:00 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 9 Oct 2017 12:03:00 -0600
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <20171009180300.GD19329@localhost.localdomain>

On Thu, Oct 05, 2017 at 12:48:23PM -0700, Chaitanya Kulkarni wrote:
> +	iod_list(req)[0] = sg_list;
> +	iod->first_dma = sgl_dma;
> +
> +	if (entries <= SGES_PER_PAGE) {
> +		nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma, entries);
> +
> +		for (i = 0; i < entries; i++) {
> +			nvme_pci_sgl_set_data(&sg_list[i], sg);
> +			length -= sg_dma_len(sg);
> +			sg = sg_next(sg);
> +		}
> +
> +		WARN_ON(length > 0);
> +		return BLK_STS_OK;
> +	}
> +
> +	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
> +
> +	do {
> +		if (i == SGES_PER_PAGE) {
> +			struct nvme_sgl_desc *old_sg_desc = sg_list;
> +			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
> +
> +			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
> +			if (!sg_list)
> +				return BLK_STS_RESOURCE;
> +
> +			i = 0;
> +			iod_list(req)[iod->npages++] = sg_list;
> +			sg_list[i++] = *link;
> +
> +			if (entries < SGES_PER_PAGE)
> +				nvme_pci_sgl_set_last_seg(link, sgl_dma, entries);
> +			else
> +				nvme_pci_sgl_set_seg(link, sgl_dma);
> +		}
> +
> +		nvme_pci_sgl_set_data(&sg_list[i], sg);
> +
> +		length -= sg_dma_len(sg);
> +		sg = sg_next(sg);
> +		entries--;
> +	} while (length > 0);

There's two loops here doing essentially the same thing, but I don't
think the single segment needs to have a special case.


From jsmart2021 at gmail.com  Mon Oct  9 13:10:32 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon,  9 Oct 2017 13:10:32 -0700
Subject: [PATCH] nvme_fc auto-connect scripts
Message-ID: <20171009201032.2018-1-jsmart2021@gmail.com>

Sagi recently posted a "one-shot" call to nvme-connect via a systemd
script. FC has no problems operating with it assuming all the required
connect fields are present (host_traddr required).

However FC is a bit more dynamic, and we've built up scripts that can
be triggered whenever a new FC remote port with a discovery subsytem is
detected. Whenever such a port is detected (by registration of the
remoteport with the nvme_fc transport by the lldd), the transport
generates a udev event.  A udev rule is created that parses the event
information into the FC remote port (traddr) that supports a discovery
subsystem and the FC host port that is connected to the FC remote port
(host_traddr). The rule then invokes an instance-based systemd service
to perform a "nvme connect-all" with the traddr and host-traddr info.

To resolve some issues seen by discovery of remote ports before the udev
service was running and where the events were not replayed, there is an
additional systemd service that runs once and invokes a script which
locates the FC host ports on the system, looks for any connected FC
remote ports with discovery subsystems, and invokes a connect-all for
them.  The FC port information is specific to lpfc attributes currently,
but when the shared transport is in place, they will be common.

I don't claim to be a systemd expert, so comments are certainly welcome.

There's a spec file for an rpm, but as a general notice, the installation
is as follows:

(as root)
cp 70-nvmefc-autoconnect.conf        /usr/lib/dracut/dracut.conf.d/
cp 70-nvmefc-autoconnect.rules       /usr/lib/udev/rules.d/
cp nvmefc-connect at .service           /usr/lib/systemd/system/
cp nvmefc-boot-connections.service   /usr/lib/systemd/system/
cp nvmefc_boot_connections.sh        /usr/sbin/lpfc/

chmod 644 /usr/lib/dracut/dracut.conf.d/70-nvmefc-autoconnect.conf
chmod 644 /usr/lib/udev/rules.d/70-nvmefc-autoconnect.rules
chmod 644 /usr/lib/systemd/system/nvmefc-connect at .service
chmod 644 /usr/lib/systemd/system/nvmefc-boot-connections.service
chmod 544 /usr/sbin/lpfc/nvmefc_boot_connections.sh

systemctl -q enable nvmefc-boot-connections
systemctl -q daemon-reload
udevadm control --reload-rules
udevadm trigger --action=change --subsystem-match=fc
dracut -f

The nvmefc_boot_connections.sh script can be put somewhere other than
/usr/sbin/lpfc. If moved, the nvmefc-boot-connections.service file must
be revised with the proper pathname.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 nvme-fc-autoconnect/70-nvmefc-autoconnect.conf     |   1 +
 nvme-fc-autoconnect/70-nvmefc-autoconnect.rules    |   6 +
 .../nvmefc-boot-connections.service                |   9 ++
 nvme-fc-autoconnect/nvmefc-connect.spec            | 138 +++++++++++++++++++++
 nvme-fc-autoconnect/nvmefc-connect at .service        |  14 +++
 nvme-fc-autoconnect/nvmefc_boot_connections.sh     |  25 ++++
 6 files changed, 193 insertions(+)
 create mode 100644 nvme-fc-autoconnect/70-nvmefc-autoconnect.conf
 create mode 100644 nvme-fc-autoconnect/70-nvmefc-autoconnect.rules
 create mode 100644 nvme-fc-autoconnect/nvmefc-boot-connections.service
 create mode 100644 nvme-fc-autoconnect/nvmefc-connect.spec
 create mode 100644 nvme-fc-autoconnect/nvmefc-connect at .service
 create mode 100644 nvme-fc-autoconnect/nvmefc_boot_connections.sh

diff --git a/nvme-fc-autoconnect/70-nvmefc-autoconnect.conf b/nvme-fc-autoconnect/70-nvmefc-autoconnect.conf
new file mode 100644
index 000000000000..b92d94fda336
--- /dev/null
+++ b/nvme-fc-autoconnect/70-nvmefc-autoconnect.conf
@@ -0,0 +1 @@
+install_items+="/usr/lib/udev/rules.d/70-nvmefc-autoconnect.rules"
diff --git a/nvme-fc-autoconnect/70-nvmefc-autoconnect.rules b/nvme-fc-autoconnect/70-nvmefc-autoconnect.rules
new file mode 100644
index 000000000000..a84ea61fca5f
--- /dev/null
+++ b/nvme-fc-autoconnect/70-nvmefc-autoconnect.rules
@@ -0,0 +1,6 @@
+#
+# nvme_fc: udev event to automatically scan (via discovery controller)
+#   new FC nvme remote ports and auto-connect to the subsystems they report.
+#
+
+ACTION=="change", SUBSYSTEM=="fc", ENV{FC_EVENT}=="nvmediscovery", ENV{NVMEFC_HOST_TRADDR}=="*",  ENV{NVMEFC_TRADDR}=="*", RUN+="/usr/bin/systemctl --no-block start nvmefc-connect at --host-traddr=$env{NVMEFC_HOST_TRADDR}\\x20--traddr=$env{NVMEFC_TRADDR}.service"
diff --git a/nvme-fc-autoconnect/nvmefc-boot-connections.service b/nvme-fc-autoconnect/nvmefc-boot-connections.service
new file mode 100644
index 000000000000..6b0a7585a43e
--- /dev/null
+++ b/nvme-fc-autoconnect/nvmefc-boot-connections.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Auto-connect to subsystems on FC-NVME devices during boot
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c "/usr/sbin/lpfc/nvmefc_boot_connections.sh"
+
+[Install]
+WantedBy=default.target
diff --git a/nvme-fc-autoconnect/nvmefc-connect.spec b/nvme-fc-autoconnect/nvmefc-connect.spec
new file mode 100644
index 000000000000..8497128096a9
--- /dev/null
+++ b/nvme-fc-autoconnect/nvmefc-connect.spec
@@ -0,0 +1,138 @@
+#/*----------------------------------------------------------------------------
+#
+# Broadcom Confidential. Copyright 2017 Broadcom. All Rights Reserved.
+# The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
+# Unpublished work. Copying, access, use, or distribution requires an
+# applicable license approved by Broadcom.
+#
+#----------------------------------------------------------------------------*/
+
+%define _sandbox        %(dirname %{_topdir})
+%define _version_str    %{_versionstr}
+
+
+Name: nvmefc-connect
+Vendor: Broadcom Ltd or its subsidiaries
+Version: %{_version_str}
+Release: 1
+Summary: NVMe FC Auto Connect
+License: Copyright 2017 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
+URL: www.broadcom.com
+Group: Applications/System
+ExclusiveOS: linux
+Requires: dracut, nvme-cli, systemd, udev
+AutoReqProv: yes
+Source: nvmefc-connect-%{version}-linux.tgz
+BuildRoot: %{_sandbox}/tmp/nvmefc-connect
+
+#During an installation
+#%pre:
+#   Runs before the package is installed
+#%post:
+#   Runs after the package is installed
+
+#During an uninstallation
+#%preun
+#   Runs before the package is uninstalled
+#%postun
+#   Runs after the package is uninstalled
+
+#During an Upgrade
+#%pre of new package
+#   Install new files
+#%post of new package
+#
+#%preun of old package
+#   Delete any old files not overwritten by newer ones
+#%postun of old package
+
+%description
+NVMe FC Auto Connect
+
+
+%prep
+
+
+
+
+%setup
+
+
+
+
+%pre
+# During an upgrade, remove the symbolic links
+if [ "$1" = "2" ]; then
+    systemctl -q disable nvmefc-boot-connections
+    systemctl -q daemon-reload
+fi
+
+
+
+
+%install
+rm -rf %{buildroot}/usr/lib/dracut/dracut.conf.d/
+rm -rf %{buildroot}/usr/lib/systemd/system/
+rm -rf %{buildroot}/usr/lib/udev/rules.d/
+rm -rf %{buildroot}/usr/sbin/lpfc/
+
+mkdir -p %{buildroot}/usr/lib/dracut/dracut.conf.d/
+mkdir -p %{buildroot}/usr/lib/systemd/system/
+mkdir -p %{buildroot}/usr/lib/udev/rules.d/
+mkdir -p %{buildroot}/usr/sbin/lpfc/
+
+cp ./usr/lib/dracut/dracut.conf.d/70-nvmefc-autoconnect.conf  %{buildroot}/usr/lib/dracut/dracut.conf.d/
+cp ./usr/lib/udev/rules.d/70-nvmefc-autoconnect.rules         %{buildroot}/usr/lib/udev/rules.d/
+cp ./usr/lib/systemd/system/nvmefc-connect at .service           %{buildroot}/usr/lib/systemd/system/
+cp ./usr/lib/systemd/system/nvmefc-boot-connections.service   %{buildroot}/usr/lib/systemd/system/
+cp ./usr/sbin/lpfc/nvmefc_boot_connections.sh                 %{buildroot}/usr/sbin/lpfc/
+
+chmod 644 %{buildroot}/usr/lib/dracut/dracut.conf.d/70-nvmefc-autoconnect.conf
+chmod 644 %{buildroot}/usr/lib/udev/rules.d/70-nvmefc-autoconnect.rules
+chmod 644 %{buildroot}/usr/lib/systemd/system/nvmefc-connect at .service
+chmod 644 %{buildroot}/usr/lib/systemd/system/nvmefc-boot-connections.service
+chmod 544 %{buildroot}/usr/sbin/lpfc/nvmefc_boot_connections.sh
+
+
+
+
+%post
+systemctl -q enable nvmefc-boot-connections
+systemctl -q daemon-reload
+udevadm control --reload-rules
+udevadm trigger --action=change --subsystem-match=fc
+dracut -f
+
+
+
+
+%preun
+# Perform the following only if this an uninstallation.
+# i.e. "rpm -ev"
+if [ "$1" = "0" ]; then
+    systemctl -q disable nvmefc-boot-connections
+    systemctl -q daemon-reload
+fi
+
+
+
+
+%postun
+systemctl -q daemon-reload
+udevadm control --reload-rules
+# Perform the following only if this an uninstallation.
+# i.e. "rpm -ev"
+if [ "$1" = "0" ]; then
+    dracut -f
+fi
+
+
+
+
+%files
+/usr/lib/dracut/dracut.conf.d/70-nvmefc-autoconnect.conf
+/usr/lib/udev/rules.d/70-nvmefc-autoconnect.rules
+/usr/lib/systemd/system/nvmefc-connect at .service
+/usr/lib/systemd/system/nvmefc-boot-connections.service
+/usr/sbin/lpfc/nvmefc_boot_connections.sh
+
diff --git a/nvme-fc-autoconnect/nvmefc-connect at .service b/nvme-fc-autoconnect/nvmefc-connect at .service
new file mode 100644
index 000000000000..c2370d63dab1
--- /dev/null
+++ b/nvme-fc-autoconnect/nvmefc-connect at .service
@@ -0,0 +1,14 @@
+#
+# Unit file used by 70-nvmefc-autoconnect.rules.
+#
+
+[Unit]
+Description=Auto-connect to subsystems on FC-NVME devices
+After=syslog.target
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c "/usr/sbin/nvme connect-all --transport=fc `/usr/bin/echo -e '%i'`"
+
+[Install]
+WantedBy=default.target
diff --git a/nvme-fc-autoconnect/nvmefc_boot_connections.sh b/nvme-fc-autoconnect/nvmefc_boot_connections.sh
new file mode 100644
index 000000000000..b1c1e67cdd8b
--- /dev/null
+++ b/nvme-fc-autoconnect/nvmefc_boot_connections.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+connect_discovery_servers_on_host()
+{
+  INFOFILE=$1/nvme_info
+  HOSTNM=`basename $1`
+  if [ -f ${INFOFILE} ] ; then
+    WWNN=`cat /sys/class/fc_host/${HOSTNM}/node_name`
+    WWPN=`cat /sys/class/fc_host/${HOSTNM}/port_name`
+    HOST_TRADDR="nn-${WWNN}:pn-${WWPN}"
+    cat ${INFOFILE} | awk -v haddr=${HOST_TRADDR} '/^NVME RPORT.*DISCSRVC/{print "/usr/sbin/nvme connect-all --transport=fc --host-traddr=" haddr " --traddr=nn-0" $6 ":pn-0" $4 }' | bash
+  fi
+}
+
+# need a real check for root permissions here
+ROOT=1
+
+if [ $ROOT -ne 1 ] ; then
+  echo "$0: Must be root to execute"
+  exit 1
+fi
+
+for f in /sys/class/scsi_host/* ; do
+  connect_discovery_servers_on_host $f
+done
-- 
2.13.1



From jsmart2021 at gmail.com  Mon Oct  9 13:13:50 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon, 9 Oct 2017 13:13:50 -0700
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <20171009201032.2018-1-jsmart2021@gmail.com>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
Message-ID: <17ab2398-17fd-1a96-3630-d841b19e0d6f@gmail.com>

Sorry - this was meant to be posted as an RFC. Please treat it as such.

-- james




From jsmart2021 at gmail.com  Mon Oct  9 13:39:44 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon,  9 Oct 2017 13:39:44 -0700
Subject: [PATCH v3] nvme_fc: fix iowait hang
Message-ID: <20171009203944.3541-1-jsmart2021@gmail.com>

Add missing iowait head initialization.
Fix irqsave vs irq: wait_event_lock_irq() doesn't do irq save/restore

Fixes: 36715cf4b366 ("nvme_fc: replace ioabort msleep loop with completion?)
Cc: <stable at vger.kernel.org> # 4.13
Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at cavium.com>
Tested-by: Himanshu Madhani <himanshu.madhani at cavium.com>
---
v3:
 no changes to patch. Added stable reference and -by's from Himanshu

 drivers/nvme/host/fc.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index b8e0822127c1..1075488a81d6 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2601,10 +2601,10 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 	nvme_fc_abort_aen_ops(ctrl);
 
 	/* wait for all io that had to be aborted */
-	spin_lock_irqsave(&ctrl->lock, flags);
+	spin_lock_irq(&ctrl->lock);
 	wait_event_lock_irq(ctrl->ioabort_wait, ctrl->iocnt == 0, ctrl->lock);
 	ctrl->flags &= ~FCCTRL_TERMIO;
-	spin_unlock_irqrestore(&ctrl->lock, flags);
+	spin_unlock_irq(&ctrl->lock);
 
 	nvme_fc_term_aen_ops(ctrl);
 
@@ -2816,6 +2816,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	ctrl->rport = rport;
 	ctrl->dev = lport->dev;
 	ctrl->cnum = idx;
+	init_waitqueue_head(&ctrl->ioabort_wait);
 
 	get_device(ctrl->dev);
 	kref_init(&ctrl->ref);
-- 
2.13.1



From rjw at rjwysocki.net  Mon Oct  9 14:52:26 2017
From: rjw at rjwysocki.net (Rafael J. Wysocki)
Date: Mon, 09 Oct 2017 23:52:26 +0200
Subject: Fwd: [Bug 196907] [Regression] New default s2idle does not work on
 Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
Message-ID: <1636342.6qveo3Fvtu@aspire.rjw.lan>

Hi,

We are seeing a resume issue with a Hynix SSD (PC300 NVMe SK hynix 512GB)
with respect to suspend-to-idle (S3 resume works for the reporter,
interestingly enough).  It appears to be limited to Hynix and to that
specific SSD ATM.

Also it is reported to work in 4.11-rc1 and it already is failing in 4.13,
so it looks like something between the two broke it.

Any ideas of what that might be?

Or anything we should try/test to narrow it down?

Any help will be appreciated.

Thanks,
Rafael
-------------- next part --------------
An embedded message was scrubbed...
From: bugzilla-daemon at bugzilla.kernel.org
Subject: [Bug 196907] [Regression] New default s2idle does not work on Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
Date: Mon, 09 Oct 2017 16:34:05 +0000
Size: 2348
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20171009/c35e6dc1/attachment.mht>

From keith.busch at intel.com  Mon Oct  9 15:14:14 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 9 Oct 2017 16:14:14 -0600
Subject: Fwd: [Bug 196907] [Regression] New default s2idle does not work
 on Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
In-Reply-To: <1636342.6qveo3Fvtu@aspire.rjw.lan>
References: <1636342.6qveo3Fvtu@aspire.rjw.lan>
Message-ID: <20171009221413.GE19329@localhost.localdomain>

On Mon, Oct 09, 2017 at 11:52:26PM +0200, Rafael J. Wysocki wrote:
> Hi,
> 
> We are seeing a resume issue with a Hynix SSD (PC300 NVMe SK hynix 512GB)
> with respect to suspend-to-idle (S3 resume works for the reporter,
> interestingly enough).  It appears to be limited to Hynix and to that
> specific SSD ATM.
> 
> Also it is reported to work in 4.11-rc1 and it already is failing in 4.13,
> so it looks like something between the two broke it.
> 
> Any ideas of what that might be?
> 
> Or anything we should try/test to narrow it down?
> 
> Any help will be appreciated.

I'd start with the APST feature and see if disabling that helps.
That can be done with the nvme_core module param:

  default_ps_max_latency_us=0


From rjw at rjwysocki.net  Mon Oct  9 15:01:33 2017
From: rjw at rjwysocki.net (Rafael J. Wysocki)
Date: Tue, 10 Oct 2017 00:01:33 +0200
Subject: Fwd: [Bug 196907] [Regression] New default s2idle does not work
 on Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
In-Reply-To: <20171009221413.GE19329@localhost.localdomain>
References: <1636342.6qveo3Fvtu@aspire.rjw.lan>
 <20171009221413.GE19329@localhost.localdomain>
Message-ID: <5479293.9hsiVanX3S@aspire.rjw.lan>

On Tuesday, October 10, 2017 12:14:14 AM CEST Keith Busch wrote:
> On Mon, Oct 09, 2017 at 11:52:26PM +0200, Rafael J. Wysocki wrote:
> > Hi,
> > 
> > We are seeing a resume issue with a Hynix SSD (PC300 NVMe SK hynix 512GB)
> > with respect to suspend-to-idle (S3 resume works for the reporter,
> > interestingly enough).  It appears to be limited to Hynix and to that
> > specific SSD ATM.
> > 
> > Also it is reported to work in 4.11-rc1 and it already is failing in 4.13,
> > so it looks like something between the two broke it.
> > 
> > Any ideas of what that might be?
> > 
> > Or anything we should try/test to narrow it down?
> > 
> > Any help will be appreciated.
> 
> I'd start with the APST feature and see if disabling that helps.
> That can be done with the nvme_core module param:
> 
>   default_ps_max_latency_us=0
> 

We've tried that and it doesn't help.



From keith.busch at intel.com  Mon Oct  9 15:22:15 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 9 Oct 2017 16:22:15 -0600
Subject: Fwd: [Bug 196907] [Regression] New default s2idle does not work
 on Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
In-Reply-To: <5479293.9hsiVanX3S@aspire.rjw.lan>
References: <1636342.6qveo3Fvtu@aspire.rjw.lan>
 <20171009221413.GE19329@localhost.localdomain>
 <5479293.9hsiVanX3S@aspire.rjw.lan>
Message-ID: <20171009222215.GF19329@localhost.localdomain>

On Tue, Oct 10, 2017 at 12:01:33AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, October 10, 2017 12:14:14 AM CEST Keith Busch wrote:
> > On Mon, Oct 09, 2017 at 11:52:26PM +0200, Rafael J. Wysocki wrote:
> > > Hi,
> > > 
> > > We are seeing a resume issue with a Hynix SSD (PC300 NVMe SK hynix 512GB)
> > > with respect to suspend-to-idle (S3 resume works for the reporter,
> > > interestingly enough).  It appears to be limited to Hynix and to that
> > > specific SSD ATM.
> > > 
> > > Also it is reported to work in 4.11-rc1 and it already is failing in 4.13,
> > > so it looks like something between the two broke it.
> > > 
> > > Any ideas of what that might be?
> > > 
> > > Or anything we should try/test to narrow it down?
> > > 
> > > Any help will be appreciated.
> > 
> > I'd start with the APST feature and see if disabling that helps.
> > That can be done with the nvme_core module param:
> > 
> >   default_ps_max_latency_us=0
> > 
> 
> We've tried that and it doesn't help.

But it wasn't done correctly according to the notes on the
bugzilla. May be just a typo, but want make sure: it says they used
"nvme.default_ps_max_latency_us=0" on the kernel's command line, which
doesn't mean anything. It's "nvme_core" that provides that module
parameter.


From rjw at rjwysocki.net  Mon Oct  9 15:13:08 2017
From: rjw at rjwysocki.net (Rafael J. Wysocki)
Date: Tue, 10 Oct 2017 00:13:08 +0200
Subject: Fwd: [Bug 196907] [Regression] New default s2idle does not work
 on Dell XPS 13 9360 with Ubuntu 16.04.3 LTS (Xenial Xerus)
In-Reply-To: <20171009222215.GF19329@localhost.localdomain>
References: <1636342.6qveo3Fvtu@aspire.rjw.lan>
 <5479293.9hsiVanX3S@aspire.rjw.lan>
 <20171009222215.GF19329@localhost.localdomain>
Message-ID: <2021794.JWb2XVz4cJ@aspire.rjw.lan>

On Tuesday, October 10, 2017 12:22:15 AM CEST Keith Busch wrote:
> On Tue, Oct 10, 2017 at 12:01:33AM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, October 10, 2017 12:14:14 AM CEST Keith Busch wrote:
> > > On Mon, Oct 09, 2017 at 11:52:26PM +0200, Rafael J. Wysocki wrote:
> > > > Hi,
> > > > 
> > > > We are seeing a resume issue with a Hynix SSD (PC300 NVMe SK hynix 512GB)
> > > > with respect to suspend-to-idle (S3 resume works for the reporter,
> > > > interestingly enough).  It appears to be limited to Hynix and to that
> > > > specific SSD ATM.
> > > > 
> > > > Also it is reported to work in 4.11-rc1 and it already is failing in 4.13,
> > > > so it looks like something between the two broke it.
> > > > 
> > > > Any ideas of what that might be?
> > > > 
> > > > Or anything we should try/test to narrow it down?
> > > > 
> > > > Any help will be appreciated.
> > > 
> > > I'd start with the APST feature and see if disabling that helps.
> > > That can be done with the nvme_core module param:
> > > 
> > >   default_ps_max_latency_us=0
> > > 
> > 
> > We've tried that and it doesn't help.
> 
> But it wasn't done correctly according to the notes on the
> bugzilla. May be just a typo, but want make sure: it says they used
> "nvme.default_ps_max_latency_us=0" on the kernel's command line, which
> doesn't mean anything. It's "nvme_core" that provides that module
> parameter.

OK, thanks!

I've added you to the CC in the BZ.

I'll ask the reporter to repeat the test with the correct param.

Thanks,
Rafael




From jsmart2021 at gmail.com  Mon Oct  9 16:39:22 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon,  9 Oct 2017 16:39:22 -0700
Subject: [PATCH v2] nvme_fc: retry initial controller connections 3 times
Message-ID: <20171009233922.7216-1-jsmart2021@gmail.com>

Currently, if a frame is lost of command fails as part of initial
association create for a new controller, the new controller connection
request will immediately fail.

Add in an immediate 3 retry loop before giving up.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2:
  Add additional comment information on why the retries. The comment
  should sufficiently explain the background.
  Note: rdma, in most cases, if initial connections or commands failed,
  would also fail on first connect attempt.
---
 drivers/nvme/host/fc.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 1075488a81d6..18e036c6833b 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2790,7 +2790,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 {
 	struct nvme_fc_ctrl *ctrl;
 	unsigned long flags;
-	int ret, idx;
+	int ret, idx, retry;
 
 	if (!(rport->remoteport.port_role &
 	    (FC_PORT_ROLE_NVME_DISCOVERY | FC_PORT_ROLE_NVME_TARGET))) {
@@ -2882,9 +2882,37 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	list_add_tail(&ctrl->ctrl_list, &rport->ctrl_list);
 	spin_unlock_irqrestore(&rport->lock, flags);
 
-	ret = nvme_fc_create_association(ctrl);
+	/*
+	 * It's possible that transactions used to create the association
+	 * may fail. Examples: CreateAssociation LS or CreateIOConnection
+	 * LS gets dropped/corrupted/fails; or a frame gets dropped or a
+	 * command times out for one of the actions to init the controller
+	 * (Connect, Get/Set_Property, Set_Features, etc). Many of these
+	 * transport errors (frame drop, LS failure) inherently must kill
+	 * the association. The transport is coded so that any command used
+	 * to create the association (prior to a LIVE state transition
+	 * while NEW or RECONNECTING) will fail if it completes in error or
+	 * times out.
+	 *
+	 * As such: as the connect request was mostly likely due to a
+	 * udev event that discovered the remote port, meaning there is
+	 * not an admin or script there to restart if the connect
+	 * request fails, retry the initial connection creation up to
+	 * three times before giving up and declaring failure.
+	 */
+	for (retry = 0; retry < 3; retry++) {
+		ret = nvme_fc_create_association(ctrl);
+		if (!ret)
+			break;
+	}
+
 	if (ret) {
+		/* couldn't schedule retry - fail out */
+		dev_err(ctrl->ctrl.device,
+			"NVME-FC{%d}: Connect retry failed\n", ctrl->cnum);
+
 		ctrl->ctrl.opts = NULL;
+
 		/* initiate nvme ctrl ref counting teardown */
 		nvme_uninit_ctrl(&ctrl->ctrl);
 		nvme_put_ctrl(&ctrl->ctrl);
-- 
2.13.1



From israelr at mellanox.com  Tue Oct 10 04:40:29 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Tue, 10 Oct 2017 14:40:29 +0300
Subject: [PATCH 0/4] Add controllers to nvmet configfs
Message-ID: <1507635633-16665-1-git-send-email-israelr@mellanox.com>

This patch series expands nvmet configfs.
It adds to nvmet two main features
 - Close an active connection from target side.
 - Show all connections from the target point of view.

Those functionalities have some benefits:
 - Give the administrator more control and information
 - Enforce new ACL while there are active connections
 - Disconnect from unreachable Hosts

Israel Rukshin (4):
  nvmet: Add port pointer to nvmet_ctrl structure
  nvmet: Fix error flow in nvmet_alloc_ctrl()
  nvmet: Add controllers to configfs
  nvmet: Add feature close connection from target side

 drivers/nvme/target/configfs.c | 86 ++++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/target/core.c     | 18 +++++++--
 drivers/nvme/target/nvmet.h    | 11 ++++++
 3 files changed, 111 insertions(+), 4 deletions(-)

-- 
1.8.4.3



From israelr at mellanox.com  Tue Oct 10 04:40:33 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Tue, 10 Oct 2017 14:40:33 +0300
Subject: [PATCH 4/4] nvmet: Add feature close connection from target side
In-Reply-To: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
Message-ID: <1507635633-16665-5-git-send-email-israelr@mellanox.com>

This allows the user to close any connection from target side
by writing to the file "force_close" at the controller folder.
Full path:
/config/nvmet/subsystems/<SUBSYSTEM_NAME>/controllers/<CTRL_ID>/force_close

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/target/configfs.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index a0942c3..cb45f6c 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -496,10 +496,23 @@ static ssize_t nvmet_ctrl_trsvcid_show(struct config_item *item, char *page)
 
 CONFIGFS_ATTR_RO(nvmet_ctrl_, trsvcid);
 
+static ssize_t nvmet_ctrl_force_close_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
+
+	ctrl->ops->delete_ctrl(ctrl);
+
+	return count;
+}
+
+CONFIGFS_ATTR_WO(nvmet_ctrl_, force_close);
+
 static struct configfs_attribute *nvmet_ctrl_attrs[] = {
 	&nvmet_ctrl_attr_hostnqn,
 	&nvmet_ctrl_attr_traddr,
 	&nvmet_ctrl_attr_trsvcid,
+	&nvmet_ctrl_attr_force_close,
 	NULL,
 };
 
-- 
1.8.4.3



From israelr at mellanox.com  Tue Oct 10 04:40:31 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Tue, 10 Oct 2017 14:40:31 +0300
Subject: [PATCH 2/4] nvmet: Fix error flow in nvmet_alloc_ctrl()
In-Reply-To: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
Message-ID: <1507635633-16665-3-git-send-email-israelr@mellanox.com>

Also rearrange nvmet_ctrl_free().

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/target/core.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 08270c1..711df73 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -798,7 +798,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 		/* Don't accept keep-alive timeout for discovery controllers */
 		if (kato) {
 			status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
-			goto out_free_sqs;
+			goto out_remove_ida;
 		}
 
 		/*
@@ -829,6 +829,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	*ctrlp = ctrl;
 	return 0;
 
+out_remove_ida:
+	ida_simple_remove(&cntlid_ida, ctrl->cntlid);
 out_free_sqs:
 	kfree(ctrl->sqs);
 out_free_cqs:
@@ -846,21 +848,22 @@ static void nvmet_ctrl_free(struct kref *ref)
 	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
 	struct nvmet_subsys *subsys = ctrl->subsys;
 
-	nvmet_stop_keep_alive_timer(ctrl);
-
 	mutex_lock(&subsys->lock);
 	list_del(&ctrl->subsys_entry);
 	mutex_unlock(&subsys->lock);
 
+	nvmet_stop_keep_alive_timer(ctrl);
+
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fatal_err_work);
 
 	ida_simple_remove(&cntlid_ida, ctrl->cntlid);
-	nvmet_subsys_put(subsys);
 
 	kfree(ctrl->sqs);
 	kfree(ctrl->cqs);
 	kfree(ctrl);
+
+	nvmet_subsys_put(subsys);
 }
 
 void nvmet_ctrl_put(struct nvmet_ctrl *ctrl)
-- 
1.8.4.3



From israelr at mellanox.com  Tue Oct 10 04:40:32 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Tue, 10 Oct 2017 14:40:32 +0300
Subject: [PATCH 3/4] nvmet: Add controllers to configfs
In-Reply-To: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
Message-ID: <1507635633-16665-4-git-send-email-israelr@mellanox.com>

The commit show all the controllers and some info about them.
This will allow the user to monitor the created controllers from target
point of view.
The "controllers" folder was added per subsystem under:
/config/nvmet/subsystems/<SUBSYSTEM_NAME>/controllers/<CTRL_ID>/

<CTRL_ID> folder consists of:
 - hostnqn: Host NQN
 - port_traddr: Port Transport Address
 - port_trsvcid: Port Transport Service ID

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/target/configfs.c | 73 ++++++++++++++++++++++++++++++++++++++++++
 drivers/nvme/target/core.c     |  6 ++++
 drivers/nvme/target/nvmet.h    | 10 ++++++
 3 files changed, 89 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index b6aeb1d..a0942c3 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -469,6 +469,49 @@ static struct config_group *nvmet_ns_make(struct config_group *group,
 	.ct_owner		= THIS_MODULE,
 };
 
+static ssize_t nvmet_ctrl_hostnqn_show(struct config_item *item, char *page)
+{
+	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->hostnqn);
+}
+
+CONFIGFS_ATTR_RO(nvmet_ctrl_, hostnqn);
+
+static ssize_t nvmet_ctrl_traddr_show(struct config_item *item, char *page)
+{
+	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->port->disc_addr.traddr);
+}
+
+CONFIGFS_ATTR_RO(nvmet_ctrl_, traddr);
+
+static ssize_t nvmet_ctrl_trsvcid_show(struct config_item *item, char *page)
+{
+	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->port->disc_addr.trsvcid);
+}
+
+CONFIGFS_ATTR_RO(nvmet_ctrl_, trsvcid);
+
+static struct configfs_attribute *nvmet_ctrl_attrs[] = {
+	&nvmet_ctrl_attr_hostnqn,
+	&nvmet_ctrl_attr_traddr,
+	&nvmet_ctrl_attr_trsvcid,
+	NULL,
+};
+
+static struct config_item_type nvmet_ctrl_type = {
+	.ct_attrs		= nvmet_ctrl_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static struct config_item_type nvmet_controllers_type = {
+	.ct_owner		= THIS_MODULE,
+};
+
 static int nvmet_port_subsys_allow_link(struct config_item *parent,
 		struct config_item *target)
 {
@@ -760,6 +803,10 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	configfs_add_default_group(&subsys->allowed_hosts_group,
 			&subsys->group);
 
+	config_group_init_type_name(&subsys->controllers_group,
+			"controllers", &nvmet_controllers_type);
+	configfs_add_default_group(&subsys->controllers_group, &subsys->group);
+
 	return &subsys->group;
 }
 
@@ -983,6 +1030,32 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
 	},
 };
 
+void nvmet_ctrl_configfs_del(struct nvmet_ctrl *ctrl)
+{
+	if (d_inode(ctrl->group.cg_item.ci_dentry))
+		configfs_unregister_group(&ctrl->group);
+}
+
+int nvmet_ctrl_configfs_create(struct nvmet_ctrl *ctrl)
+{
+	int res = 0;
+	char name[CONFIGFS_ITEM_NAME_LEN];
+
+	sprintf(name, "%d", ctrl->cntlid);
+	pr_info("Adding controller %s to configfs\n", name);
+
+	config_group_init_type_name(&ctrl->group, name, &nvmet_ctrl_type);
+
+	res = configfs_register_group(&ctrl->subsys->controllers_group,
+				      &ctrl->group);
+
+	if (res)
+		pr_err("failed to register configfs group for controller %s\n",
+		       name);
+
+	return res;
+}
+
 int __init nvmet_init_configfs(void)
 {
 	int ret;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 711df73..99bbf49 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -820,6 +820,10 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 		ctrl->kato = DIV_ROUND_UP(kato, 1000);
 	}
 	ctrl->port = req->port;
+	ret = nvmet_ctrl_configfs_create(ctrl);
+	if (ret)
+		goto out_remove_ida;
+
 	nvmet_start_keep_alive_timer(ctrl);
 
 	mutex_lock(&subsys->lock);
@@ -854,6 +858,8 @@ static void nvmet_ctrl_free(struct kref *ref)
 
 	nvmet_stop_keep_alive_timer(ctrl);
 
+	nvmet_ctrl_configfs_del(ctrl);
+
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fatal_err_work);
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index dc586e7..1d49873 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -135,8 +135,15 @@ struct nvmet_ctrl {
 
 	char			subsysnqn[NVMF_NQN_FIELD_LEN];
 	char			hostnqn[NVMF_NQN_FIELD_LEN];
+
+	struct config_group	group;
 };
 
+static inline struct nvmet_ctrl *to_nvmet_ctrl(struct config_item *item)
+{
+	return container_of(to_config_group(item), struct nvmet_ctrl, group);
+}
+
 struct nvmet_subsys {
 	enum nvme_subsys_type	type;
 
@@ -161,6 +168,7 @@ struct nvmet_subsys {
 
 	struct config_group	namespaces_group;
 	struct config_group	allowed_hosts_group;
+	struct config_group	controllers_group;
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -285,6 +293,8 @@ u16 nvmet_ctrl_find_get(const char *subsysnqn, const char *hostnqn, u16 cntlid,
 		struct nvmet_req *req, struct nvmet_ctrl **ret);
 void nvmet_ctrl_put(struct nvmet_ctrl *ctrl);
 u16 nvmet_check_ctrl_status(struct nvmet_req *req, struct nvme_command *cmd);
+int nvmet_ctrl_configfs_create(struct nvmet_ctrl *ctrl);
+void nvmet_ctrl_configfs_del(struct nvmet_ctrl *ctrl);
 
 struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 		enum nvme_subsys_type type);
-- 
1.8.4.3



From israelr at mellanox.com  Tue Oct 10 04:40:30 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Tue, 10 Oct 2017 14:40:30 +0300
Subject: [PATCH 1/4] nvmet: Add port pointer to nvmet_ctrl structure
In-Reply-To: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
Message-ID: <1507635633-16665-2-git-send-email-israelr@mellanox.com>

The ctrl is the "engine" of the subsystem per port, so link them
for future usage.

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/target/core.c  | 1 +
 drivers/nvme/target/nvmet.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 1b208be..08270c1 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -819,6 +819,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 		/* keep-alive timeout in seconds */
 		ctrl->kato = DIV_ROUND_UP(kato, 1000);
 	}
+	ctrl->port = req->port;
 	nvmet_start_keep_alive_timer(ctrl);
 
 	mutex_lock(&subsys->lock);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 7b8e20a..dc586e7 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -108,6 +108,7 @@ static inline struct nvmet_port *to_nvmet_port(struct config_item *item)
 
 struct nvmet_ctrl {
 	struct nvmet_subsys	*subsys;
+	struct nvmet_port	*port;
 	struct nvmet_cq		**cqs;
 	struct nvmet_sq		**sqs;
 
-- 
1.8.4.3



From jthumshirn at suse.de  Tue Oct 10 06:48:44 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Tue, 10 Oct 2017 15:48:44 +0200
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <20171009201032.2018-1-jsmart2021@gmail.com>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
Message-ID: <20171010134844.hlgn7mif6yek2vvp@linux-x5ow.site>

On Mon, Oct 09, 2017 at 01:10:32PM -0700, James Smart wrote:
> cp nvmefc_boot_connections.sh        /usr/sbin/lpfc/

[...]

> +ExecStart=/bin/sh -c "/usr/sbin/lpfc/nvmefc_boot_connections.sh"

/usr/sbin/lpfc wont get past our package reviewers, %_sbindir (for rpm, I'm
sure there is an equivalent for deb as well) should be used there.


Also wouldn't it make sense to have the scripts/services in nvme-cli?

Byte,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From roys at lightbitslabs.com  Tue Oct 10 08:59:27 2017
From: roys at lightbitslabs.com (roys at lightbitslabs.com)
Date: Tue, 10 Oct 2017 18:59:27 +0300
Subject: [PATCH] nvme/target: Change max_nsid in subsystem due to ns_disable
 if needed
Message-ID: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>

From: Roy Shterman <roys at lightbitslabs.com>

In case we disable namespaces which has the nsid like
subsystem max_nsid we need to search for the next largest nsid
in this subsystem. If the subsystem don't has more namespaces
we set it to 0, else we take nsid from the last namespace in
namespaces list because the list is sorted while inserting.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
---
 drivers/nvme/target/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index eb9399a..e996972 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -334,6 +334,17 @@ void nvmet_ns_disable(struct nvmet_ns *ns)
 
 	ns->enabled = false;
 	list_del_rcu(&ns->dev_link);
+
+	if (ns->nsid == subsys->max_nsid) {
+		if (list_empty(&subsys->namespaces)) {
+			subsys->max_nsid = 0;
+		} else {
+			struct nvmet_ns *ns = list_last_entry(&subsys->namespaces, struct nvmet_ns, entry);
+
+			subsys->max_nsid = ns->nsid;
+		}
+	}
+
 	mutex_unlock(&subsys->lock);
 
 	/*
-- 
2.7.4



From greg at kroah.com  Tue Oct 10 10:47:20 2017
From: greg at kroah.com (Greg KH)
Date: Tue, 10 Oct 2017 19:47:20 +0200
Subject: [PATCH-stable] nvme: protect against simultaneous shutdown
 invocations
In-Reply-To: <20171006151455.18816-1-keith.busch@intel.com>
References: <20171006151455.18816-1-keith.busch@intel.com>
Message-ID: <20171010174720.GA747@kroah.com>

On Fri, Oct 06, 2017 at 09:14:55AM -0600, Keith Busch wrote:
> commit 77bf25ea70200cddf083f74b7f617e5f07fac8bd upstream
> 
> Back-ported to 4.4. The difference is the file location of the struct
> definition that's adding the mutex.
> 
> This fixes reported kernel panics in 4.4-stable from simultaneous
> controller resets that was never supposed to be allowed to happen.
> 
> Cc: <stable at vger.kernel.org> # 4.4
> Signed-off-by: Keith Busch <keith.busch at intel.com>
> ---
>  drivers/nvme/host/nvme.h | 2 ++
>  drivers/nvme/host/pci.c  | 3 +++
>  2 files changed, 5 insertions(+)

Thanks for the patch, now queued up.

greg k-h


From jsmart2021 at gmail.com  Tue Oct 10 17:35:11 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 10 Oct 2017 17:35:11 -0700
Subject: [PATCH 2/3] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171011003512.5946-1-jsmart2021@gmail.com>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
Message-ID: <20171011003512.5946-3-jsmart2021@gmail.com>

Adds support for the duplicate_connect option. When set to true,
checks whether there's an existing controller via the same host port
and target port for the same host (hostnqn, hostid) to the same
subsystem. Fails the connection request if an existing controller.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 18e036c6833b..48c241620d2b 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2784,6 +2784,46 @@ static const struct blk_mq_ops nvme_fc_admin_mq_ops = {
 };
 
 
+static inline bool
+__nvme_fc_options_match(struct nvmf_ctrl_options *opts,
+			struct nvme_fc_ctrl *ctrl)
+{
+	if (strcmp(opts->subsysnqn, ctrl->ctrl.opts->subsysnqn) ||
+	    strcmp(opts->host->nqn, ctrl->ctrl.opts->host->nqn) ||
+	    memcmp(&opts->host->id, &ctrl->ctrl.opts->host->id,
+				sizeof(uuid_t)))
+		return false;
+
+	return true;
+}
+
+/*
+ * Fails a controller request if it matches an existing controller
+ * (association) with the same tuple:
+ * <Host NQN, Host ID, local FC port, remote FC port, SUBSYS NQN>
+ *
+ * The ports don't need to be compared as they are intrinsically
+ * already matched by the port pointers supplied.
+ */
+static bool
+nvme_fc_existing_controller(struct nvme_fc_rport *rport,
+		struct nvmf_ctrl_options *opts)
+{
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+	bool found = false;
+
+	spin_lock_irqsave(&rport->lock, flags);
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
+		found = __nvme_fc_options_match(opts, ctrl);
+		if (found)
+			break;
+	}
+	spin_unlock_irqrestore(&rport->lock, flags);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	struct nvme_fc_lport *lport, struct nvme_fc_rport *rport)
@@ -2798,6 +2838,12 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 		goto out_fail;
 	}
 
+	if (!opts->duplicate_connect &&
+	    nvme_fc_existing_controller(rport, opts)) {
+		ret = -EALREADY;
+		goto out_fail;
+	}
+
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl) {
 		ret = -ENOMEM;
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 10 17:35:09 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 10 Oct 2017 17:35:09 -0700
Subject: [PATCH 0/3] Add duplicate_connect option for ctlr connect requests
Message-ID: <20171011003512.5946-1-jsmart2021@gmail.com>

This patch adds a duplicate_connect option option to those supported by
fabric connection requests. When set to true (1), duplicate connections
are allowed. When set to false (0), duplicate connections are rejected.

The option is set in the fabrics layer and acted upon by each transport.

This patch set came about as of this prior patch review:
http://lists.infradead.org/pipermail/linux-nvme/2017-September/012783.html

James Smart (3):
  nvme: add duplicate_connect option
  nvme_fc: add support for duplicate_connect option
  RFC: nvme rdma: add support for duplicate_connect option

 drivers/nvme/host/fabrics.c | 19 +++++++++-
 drivers/nvme/host/fabrics.h |  2 ++
 drivers/nvme/host/fc.c      | 46 +++++++++++++++++++++++++
 drivers/nvme/host/rdma.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 150 insertions(+), 1 deletion(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 10 17:35:12 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 10 Oct 2017 17:35:12 -0700
Subject: [PATCH/RFC 3/3] nvme rdma: add support for duplicate_connect option
In-Reply-To: <20171011003512.5946-1-jsmart2021@gmail.com>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
Message-ID: <20171011003512.5946-4-jsmart2021@gmail.com>

Adds support for the duplicate_connect option. When set to true,
checks whether there's an existing controller via the same target
address (traddr), target port (trsvcid), and if specified, host
address (host_traddr). Fails the connection request if there is
an existing controller.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/rdma.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..b5f45d64cf73 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1847,6 +1847,85 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.get_address		= nvmf_get_address,
 };
 
+static inline bool
+__nvme_rdma_options_match(struct nvme_rdma_ctrl *ctrl,
+	struct nvmf_ctrl_options *opts)
+{
+	char *stdport = __stringify(NVME_RDMA_IP_PORT);
+
+	if (strcmp(opts->subsysnqn, ctrl->ctrl.opts->subsysnqn) ||
+	    strcmp(opts->host->nqn, ctrl->ctrl.opts->host->nqn) ||
+	    memcmp(&opts->host->id, &ctrl->ctrl.opts->host->id,
+				sizeof(uuid_t)) ||
+	    strcmp(opts->traddr, ctrl->ctrl.opts->traddr))
+		return false;
+
+	if (opts->mask & NVMF_OPT_TRSVCID &&
+	    ctrl->ctrl.opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(opts->trsvcid, ctrl->ctrl.opts->trsvcid))
+			return false;
+	} else if (opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(opts->trsvcid, stdport))
+			return false;
+	} else if (ctrl->ctrl.opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(stdport, ctrl->ctrl.opts->trsvcid))
+			return false;
+	}
+	/* else, it's a match as both have stdport. Fall to next checks */
+
+	/*
+	 * checking the local address is rough. In most cases, one
+	 * is not specified and the host port is selected by the stack.
+	 *
+	 * Assume no match if:
+	 *  local address is specified and address is not the same
+	 *  local address is not specified but remote is, or vice versa
+	 *    (admin using specific host_traddr when it matters).
+	 */
+	if (opts->mask & NVMF_OPT_HOST_TRADDR &&
+	    ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR) {
+		if (strcmp(opts->host_traddr, ctrl->ctrl.opts->host_traddr))
+			return false;
+	} else if (opts->mask & NVMF_OPT_HOST_TRADDR ||
+		   ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR)
+		return false;
+	/*
+	 * if neither controller had an host port specified, assume it's
+	 * a match as everything else matched.
+	 */
+
+	return true;
+}
+
+/*
+ * Fails a connection request if it matches an existing controller
+ * (association) with the same tuple:
+ * <Host NQN, Host ID, local address, remote address, remote port, SUBSYS NQN>
+ *
+ * if local address is not specified in the request, it will match an
+ * existing controller with all the other parameters the same and no
+ * local port address specified as well.
+ *
+ * The ports don't need to be compared as they are intrinsically
+ * already matched by the port pointers supplied.
+ */
+static bool
+nvme_rdma_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_rdma_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_rdma_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_rdma_ctrl_list, list) {
+		found = __nvme_rdma_options_match(ctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_rdma_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
@@ -1883,6 +1962,11 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 		}
 	}
 
+	if (!opts->duplicate_connect && nvme_rdma_existing_controller(opts)) {
+		ret = -EALREADY;
+		goto out_free_ctrl;
+	}
+
 	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops,
 				0 /* no quirks, we're perfect! */);
 	if (ret)
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 10 17:35:10 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 10 Oct 2017 17:35:10 -0700
Subject: [PATCH 1/3] nvme: add duplicate_connect option
In-Reply-To: <20171011003512.5946-1-jsmart2021@gmail.com>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
Message-ID: <20171011003512.5946-2-jsmart2021@gmail.com>

Add the option "duplicate_connect", which can be 1 for true, 0 for false.
Default is false.

When false, the transport should validate whether a new controller request
is targeted for the same host transport addressing and target transport
addressing as an existing controller. If so, the new controller request
should be rejected.

When true, the callee is explicitly requesting a duplicate controller
connection to be made and the new request should be attempted.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fabrics.c | 19 ++++++++++++++++++-
 drivers/nvme/host/fabrics.h |  2 ++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 8bca36a46924..4177137a2ee6 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -548,6 +548,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_HOSTNQN,		"hostnqn=%s"		},
 	{ NVMF_OPT_HOST_TRADDR,		"host_traddr=%s"	},
 	{ NVMF_OPT_HOST_ID,		"hostid=%s"		},
+	{ NVMF_OPT_DUP_CONNECT,		"duplicate_connect=%d"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -566,6 +567,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	opts->nr_io_queues = num_online_cpus();
 	opts->reconnect_delay = NVMF_DEF_RECONNECT_DELAY;
 	opts->kato = NVME_DEFAULT_KATO;
+	opts->duplicate_connect = false;
 
 	options = o = kstrdup(buf, GFP_KERNEL);
 	if (!options)
@@ -742,6 +744,21 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 				goto out;
 			}
 			break;
+		case NVMF_OPT_DUP_CONNECT:
+			if (match_int(args, &token)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (token == 0)
+				opts->duplicate_connect = false;
+			else if (token == 1)
+				opts->duplicate_connect = true;
+			else {
+				pr_err("Invalid duplicate_connect %d\n", token);
+				ret = -EINVAL;
+				goto out;
+			}
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -823,7 +840,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 #define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
 #define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID)
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index bf33663218cd..a7590cc59ba2 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -57,6 +57,7 @@ enum {
 	NVMF_OPT_HOST_TRADDR	= 1 << 10,
 	NVMF_OPT_CTRL_LOSS_TMO	= 1 << 11,
 	NVMF_OPT_HOST_ID	= 1 << 12,
+	NVMF_OPT_DUP_CONNECT	= 1 << 13,
 };
 
 /**
@@ -96,6 +97,7 @@ struct nvmf_ctrl_options {
 	unsigned int		nr_io_queues;
 	unsigned int		reconnect_delay;
 	bool			discovery_nqn;
+	bool			duplicate_connect;
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 10 17:35:36 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 10 Oct 2017 17:35:36 -0700
Subject: [PATCH] nvme-cli: add duplicate_connect parameter support
Message-ID: <20171011003536.5996-1-jsmart2021@gmail.com>

Add support for the duplicate_connect parameter for the fabrics
device.

Rather than add yet another 6-line code block, created a small helper
routine invoked for each argument option.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 fabrics.c | 124 +++++++++++++++++++-------------------------------------------
 1 file changed, 38 insertions(+), 86 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index 4871e17..dcde6c2 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -59,6 +59,7 @@ static struct config {
 	char *keep_alive_tmo;
 	char *reconnect_delay;
 	char *ctrl_loss_tmo;
+	char *duplicate_connect;
 	char *raw;
 	char *device;
 } cfg = { NULL };
@@ -484,6 +485,22 @@ out:
 	return ret;
 }
 
+static int
+add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
+{
+	int len;
+
+	if (arg) {
+		len = snprintf(*argstr, *max_len, ",%s=%s", arg_str, arg);
+		if (len < 0)
+			return -EINVAL;
+		*argstr += len;
+		*max_len -= len;
+	}
+
+	return 0;
+}
+
 static int build_options(char *argstr, int max_len)
 {
 	int len;
@@ -500,99 +517,33 @@ static int build_options(char *argstr, int max_len)
 		}
 	}
 
+	/* always specify nqn as first arg - this will init the string */
 	len = snprintf(argstr, max_len, "nqn=%s", cfg.nqn);
 	if (len < 0)
 		return -EINVAL;
 	argstr += len;
 	max_len -= len;
 
-	len = snprintf(argstr, max_len, ",transport=%s", cfg.transport);
-	if (len < 0)
+	if (add_argument(&argstr, &max_len, "transport", cfg.transport) ||
+	    add_argument(&argstr, &max_len, "traddr", cfg.traddr) ||
+	    add_argument(&argstr, &max_len, "host_traddr", cfg.host_traddr) ||
+	    add_argument(&argstr, &max_len, "trsvcid", cfg.trsvcid) ||
+	    ((cfg.hostnqn || nvmf_hostnqn_file()) &&
+		    add_argument(&argstr, &max_len, "hostnqn", cfg.hostnqn)) ||
+	    ((cfg.hostid || nvmf_hostid_file()) &&
+		    add_argument(&argstr, &max_len, "hostid", cfg.hostid)) ||
+	    add_argument(&argstr, &max_len, "nr_io_queues",
+		    cfg.nr_io_queues) ||
+	    add_argument(&argstr, &max_len, "queue_size", cfg.queue_size) ||
+	    add_argument(&argstr, &max_len, "keep_alive_tmo",
+		    cfg.keep_alive_tmo) ||
+	    add_argument(&argstr, &max_len, "reconnect_delay",
+		    cfg.reconnect_delay) ||
+	    add_argument(&argstr, &max_len, "ctrl_loss_tmo",
+		    cfg.ctrl_loss_tmo) ||
+	    add_argument(&argstr, &max_len, "duplicate_connect",
+		    cfg.duplicate_connect))
 		return -EINVAL;
-	argstr += len;
-	max_len -= len;
-
-	if (cfg.traddr) {
-		len = snprintf(argstr, max_len, ",traddr=%s", cfg.traddr);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.host_traddr) {
-		len = snprintf(argstr, max_len, ",host_traddr=%s", cfg.host_traddr);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.trsvcid) {
-		len = snprintf(argstr, max_len, ",trsvcid=%s", cfg.trsvcid);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.hostnqn || nvmf_hostnqn_file()) {
-		len = snprintf(argstr, max_len, ",hostnqn=%s", cfg.hostnqn);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.hostid || nvmf_hostid_file()) {
-		len = snprintf(argstr, max_len, ",hostid=%s", cfg.hostid);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.nr_io_queues) {
-		len = snprintf(argstr, max_len, ",nr_io_queues=%s",
-				cfg.nr_io_queues);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.queue_size) {
-		len = snprintf(argstr, max_len, ",queue_size=%s",
-				cfg.queue_size);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.keep_alive_tmo) {
-		len = snprintf(argstr, max_len, ",keep_alive_tmo=%s", cfg.keep_alive_tmo);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.reconnect_delay) {
-		len = snprintf(argstr, max_len, ",reconnect_delay=%s", cfg.reconnect_delay);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.ctrl_loss_tmo) {
-		len = snprintf(argstr, max_len, ",ctrl_loss_tmo=%s", cfg.ctrl_loss_tmo);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
 
 	return 0;
 }
@@ -873,6 +824,7 @@ int connect(const char *desc, int argc, char **argv)
 		{"keep-alive-tmo",  'k', "LIST", CFG_STRING, &cfg.keep_alive_tmo,  required_argument, "keep alive timeout period in seconds" },
 		{"reconnect-delay", 'c', "LIST", CFG_STRING, &cfg.reconnect_delay, required_argument, "reconnect timeout period in seconds" },
 		{"ctrl-loss-tmo",   'l', "LIST", CFG_STRING, &cfg.ctrl_loss_tmo,   required_argument, "controller loss timeout period in seconds" },
+		{"duplicate_connect", 'D', "LIST", CFG_STRING, &cfg.duplicate_connect, required_argument, "allow duplicate connections between same transport host and subsystem port" },
 		{NULL},
 	};
 
-- 
2.13.1



From lv.zheng at intel.com  Tue Oct 10 23:16:10 2017
From: lv.zheng at intel.com (Zheng, Lv)
Date: Wed, 11 Oct 2017 06:16:10 +0000
Subject: ACPI object counter overrun on frequent nvme resets
In-Reply-To: <1AE640813FDE7649BE1B193DEA596E886CF6F735@SHSMSX101.ccr.corp.intel.com>
References: <20170925130729.GA14151@infradead.org>
 <1AE640813FDE7649BE1B193DEA596E886CF6F735@SHSMSX101.ccr.corp.intel.com>
Message-ID: <1AE640813FDE7649BE1B193DEA596E886CF7A228@SHSMSX101.ccr.corp.intel.com>

> From: Zheng, Lv
> Subject: RE: ACPI object counter overrun on frequent nvme resets
> 
> Hi
> 
> I've converted this bug into a Bugzilla entry here:
> https://bugzilla.kernel.org/show_bug.cgi?id=197071

Christoph, could you help to provide the detailed debugging information there.
We need dmesg output and acpidump output.

Thanks in advance.

> 
> Thanks
> Lv
> 
> > From: linux-acpi-owner at vger.kernel.org [mailto:linux-acpi-owner at vger.kernel.org] On Behalf Of
> > Subject: ACPI object counter overrun on frequent nvme resets
> >
> > Hi all,
> >
> > when doing an error injection test with the nvme-pci driver on Linux 4.14
> > almost rc2 (820bf5c419e4b85298e5c3001bd1b5be46d60765 plus a few nvme
> > patches) that does a lot of nvme resets which shut down and re-enable the
> > PCI device I see a flood of warnings like:
> >
> > [   78.237286] ACPI Warning: Large Reference Count (0x1001) in object ffff88007d702c68, Type=0x1D
> > (20170728/utdelete-473)
> >
> > Throwing in a dump_stack into the path it comes from:
> >
> > [   78.238387] CPU: 0 PID: 286 Comm: kworker/u8:3 Not tainted 4.14.0-rc1+ #134
> > [   78.238998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
> > [   78.239720] Workqueue: nvme-wq nvme_reset_work
> > [   78.239995] Call Trace:
> > [   78.239995]  dump_stack+0x63/0x83
> > [   78.239995]  acpi_ut_update_ref_count+0x2f7/0x2fe
> > [   78.239995]  acpi_ut_update_object_reference+0x114/0x182
> > [   78.239995]  acpi_ut_add_reference+0x1b/0x1e
> > [   78.239995]  acpi_ds_init_package_element+0x149/0x16e
> > [   78.239995]  ? acpi_ds_build_internal_object+0xca/0x12c
> > [   78.239995]  acpi_ds_build_internal_package_obj+0x1b2/0x258
> > [   78.239995]  ? acpi_ds_build_internal_package_obj+0x1b2/0x258
> > [   78.239995]  acpi_ds_eval_data_object_operands+0xd3/0x123
> > [   78.239995]  acpi_ds_exec_end_op+0x288/0x3e1
> > [   78.239995]  acpi_ps_parse_loop+0x519/0x57b
> > [   78.239995]  acpi_ps_parse_aml+0x93/0x29c
> > [   78.239995]  acpi_ps_execute_method+0x148/0x17f
> > [   78.239995]  acpi_ns_evaluate+0x1c1/0x24d
> > [   78.239995]  acpi_ut_evaluate_object+0x72/0x197
> > [   78.239995]  acpi_rs_get_prt_method_data+0x21/0x45
> > [   78.239995]  acpi_get_irq_routing_table+0x2c/0x30
> > [   78.239995]  acpi_pci_irq_find_prt_entry+0x8e/0x2c0
> > [   78.239995]  ? irq_get_irq_data+0x9/0x20
> > [   78.239995]  ? mp_unmap_irq+0xf/0x70
> > [   78.239995]  acpi_pci_irq_lookup+0x26/0x1a0
> > [   78.239995]  acpi_pci_irq_enable+0x5b/0x1b0
> > [   78.239995]  ? pci_read_config_word+0x22/0x30
> > [   78.239995]  pcibios_enable_device+0x28/0x30
> > [   78.239995]  do_pci_enable_device+0x5f/0xe0
> > [   78.239995]  pci_enable_device_flags+0xc3/0x110
> > [   78.239995]  pci_enable_device_mem+0xe/0x10
> > [   78.239995]  nvme_reset_work+0x4c/0x1610
> > [   78.239995]  ? sched_clock+0x9/0x10
> > [   78.239995]  ? sched_clock_local+0x17/0x90
> > [   78.239995]  ? _raw_spin_lock+0x9/0x10
> > [   78.239995]  ? pick_next_task_fair+0x420/0x690
> > [   78.239995]  ? _raw_spin_unlock_irq+0x9/0x20
> > [   78.239995]  ? finish_task_switch+0x7b/0x1f0
> > [   78.239995]  ? __schedule+0x2d7/0x800
> > [   78.239995]  process_one_work+0x1db/0x3e0
> > [   78.239995]  worker_thread+0x43/0x3f0
> > [   78.239995]  kthread+0x103/0x140
> > [   78.239995]  ? process_one_work+0x3e0/0x3e0
> > [   78.239995]  ? kthread_create_on_node+0x40/0x40
> > [   78.239995]  ret_from_fork+0x25/0x30
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html


From sagi at grimberg.me  Wed Oct 11 02:49:51 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 12:49:51 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: stop controller reset if the controller is
 deleting
Message-ID: <1507715391-430-1-git-send-email-sagi@grimberg.me>

If the controller is deleting (in case the user decided to delete
it), we have no point to continue reset sequence.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 039f38cb6987..cf2ac5d893d5 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1821,7 +1821,11 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
 	}
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
-	WARN_ON_ONCE(!changed);
+	if (!changed) {
+		/* state change failure is ok if we're in DELETING state */
+		WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
+		return;
+	}
 
 	nvme_start_ctrl(&ctrl->ctrl);
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 02:53:05 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 12:53:05 +0300
Subject: [PATCH v2 0/3] Move tagset reinit to its only current consumer,
 nvme-core
Message-ID: <1507715588-1041-1-git-send-email-sagi@grimberg.me>

As suggested by Bart Van Assche, get rid of
blk_mq_reinit_tagset and move it to nvme-core (its
only current consumer).

Instead, introduce a more generic tagset iterator helper.

Changes from v1:
- pass tagset to nvme_reinit_tagset (Bart)

Sagi Grimberg (3):
  block: introduce blk_mq_tagset_iter
  nvme: introduce nvme_reinit_tagset
  block: remove blk_mq_reinit_tagset

 block/blk-mq-tag.c       | 11 +++++------
 drivers/nvme/host/core.c | 10 ++++++++++
 drivers/nvme/host/fc.c   |  3 ++-
 drivers/nvme/host/nvme.h |  2 ++
 drivers/nvme/host/rdma.c |  7 +++----
 include/linux/blk-mq.h   |  4 ++--
 6 files changed, 24 insertions(+), 13 deletions(-)

-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 02:53:06 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 12:53:06 +0300
Subject: [PATCH v2 1/3] block: introduce blk_mq_tagset_iter
In-Reply-To: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507715588-1041-2-git-send-email-sagi@grimberg.me>

Iterator helper to apply a function on all the
tags in a given tagset. export it as it will be used
outside the block layer later on.

Reviewed-by: Bart Van Assche <bart.vanassche at wdc.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 block/blk-mq-tag.c     | 16 +++++++++++-----
 include/linux/blk-mq.h |  2 ++
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 6714507aa6c7..bce1c76fc768 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -298,12 +298,12 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,
 }
 EXPORT_SYMBOL(blk_mq_tagset_busy_iter);
 
-int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
-			 int (reinit_request)(void *, struct request *))
+int blk_mq_tagset_iter(struct blk_mq_tag_set *set, void *data,
+			 int (fn)(void *, struct request *))
 {
 	int i, j, ret = 0;
 
-	if (WARN_ON_ONCE(!reinit_request))
+	if (WARN_ON_ONCE(!fn))
 		goto out;
 
 	for (i = 0; i < set->nr_hw_queues; i++) {
@@ -316,8 +316,7 @@ int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
 			if (!tags->static_rqs[j])
 				continue;
 
-			ret = reinit_request(set->driver_data,
-					     tags->static_rqs[j]);
+			ret = fn(data, tags->static_rqs[j]);
 			if (ret)
 				goto out;
 		}
@@ -326,6 +325,13 @@ int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
 out:
 	return ret;
 }
+EXPORT_SYMBOL_GPL(blk_mq_tagset_iter);
+
+int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
+			 int (reinit_request)(void *, struct request *))
+{
+	return blk_mq_tagset_iter(set, set->driver_data, reinit_request);
+}
 EXPORT_SYMBOL_GPL(blk_mq_reinit_tagset);
 
 void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn,
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 50c6485cb04f..6bc29f73a9aa 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -259,6 +259,8 @@ void blk_freeze_queue_start(struct request_queue *q);
 void blk_mq_freeze_queue_wait(struct request_queue *q);
 int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout);
+int blk_mq_tagset_iter(struct blk_mq_tag_set *set, void *data,
+		int (reinit_request)(void *, struct request *));
 int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
 			 int (reinit_request)(void *, struct request *));
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 02:53:07 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 12:53:07 +0300
Subject: [PATCH v2 2/3] nvme: introduce nvme_reinit_tagset
In-Reply-To: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507715588-1041-3-git-send-email-sagi@grimberg.me>

Move blk_mq_reinit_tagset from blk-mq to nvme core
as the only user of it. Current transports that use
it (rdma, fc) simply implement .reinit_request op.

This patch does not change any functionality.

Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 10 ++++++++++
 drivers/nvme/host/fc.c   |  3 ++-
 drivers/nvme/host/nvme.h |  2 ++
 drivers/nvme/host/rdma.c |  7 +++----
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 26c8913435b2..7414e25a75f0 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2944,6 +2944,16 @@ void nvme_start_queues(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_start_queues);
 
+int nvme_reinit_tagset(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set)
+{
+	if (!ctrl->ops->reinit_request)
+		return 0;
+
+	return blk_mq_tagset_iter(set, set->driver_data,
+			ctrl->ops->reinit_request);
+}
+EXPORT_SYMBOL_GPL(nvme_reinit_tagset);
+
 int __init nvme_core_init(void)
 {
 	int result;
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index b8e0822127c1..372a0685b12a 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2393,7 +2393,7 @@ nvme_fc_reinit_io_queues(struct nvme_fc_ctrl *ctrl)
 
 	nvme_fc_init_io_queues(ctrl);
 
-	ret = blk_mq_reinit_tagset(&ctrl->tag_set, nvme_fc_reinit_request);
+	ret = nvme_reinit_tagset(&ctrl->ctrl, ctrl->ctrl.tagset);
 	if (ret)
 		goto out_free_io_queues;
 
@@ -2753,6 +2753,7 @@ static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
 	.submit_async_event	= nvme_fc_submit_async_event,
 	.delete_ctrl		= nvme_fc_del_nvme_ctrl,
 	.get_address		= nvmf_get_address,
+	.reinit_request		= nvme_fc_reinit_request,
 };
 
 static void
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index df787f39f4c1..cb9d93048f3d 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -237,6 +237,7 @@ struct nvme_ctrl_ops {
 	void (*submit_async_event)(struct nvme_ctrl *ctrl, int aer_idx);
 	int (*delete_ctrl)(struct nvme_ctrl *ctrl);
 	int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
+	int (*reinit_request)(void *data, struct request *rq);
 };
 
 static inline bool nvme_ctrl_ready(struct nvme_ctrl *ctrl)
@@ -311,6 +312,7 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl);
 void nvme_wait_freeze(struct nvme_ctrl *ctrl);
 void nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout);
 void nvme_start_freeze(struct nvme_ctrl *ctrl);
+int nvme_reinit_tagset(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set);
 
 #define NVME_QID_ANY -1
 struct request *nvme_alloc_request(struct request_queue *q,
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..039f38cb6987 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -774,8 +774,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 			goto out_free_tagset;
 		}
 	} else {
-		error = blk_mq_reinit_tagset(&ctrl->admin_tag_set,
-					     nvme_rdma_reinit_request);
+		error = nvme_reinit_tagset(&ctrl->ctrl, ctrl->ctrl.admin_tagset);
 		if (error)
 			goto out_free_queue;
 	}
@@ -855,8 +854,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 			goto out_free_tag_set;
 		}
 	} else {
-		ret = blk_mq_reinit_tagset(&ctrl->tag_set,
-					   nvme_rdma_reinit_request);
+		ret = nvme_reinit_tagset(&ctrl->ctrl, ctrl->ctrl.tagset);
 		if (ret)
 			goto out_free_io_queues;
 
@@ -1845,6 +1843,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.submit_async_event	= nvme_rdma_submit_async_event,
 	.delete_ctrl		= nvme_rdma_del_ctrl,
 	.get_address		= nvmf_get_address,
+	.reinit_request		= nvme_rdma_reinit_request,
 };
 
 static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 02:53:08 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 12:53:08 +0300
Subject: [PATCH v2 3/3] block: remove blk_mq_reinit_tagset
In-Reply-To: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507715588-1041-4-git-send-email-sagi@grimberg.me>

No callers left.

Reviewed-by: Bart Van Assche <bart.vanassche at wdc.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 block/blk-mq-tag.c     | 7 -------
 include/linux/blk-mq.h | 2 --
 2 files changed, 9 deletions(-)

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index bce1c76fc768..c81b40ecd3f1 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -327,13 +327,6 @@ int blk_mq_tagset_iter(struct blk_mq_tag_set *set, void *data,
 }
 EXPORT_SYMBOL_GPL(blk_mq_tagset_iter);
 
-int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
-			 int (reinit_request)(void *, struct request *))
-{
-	return blk_mq_tagset_iter(set, set->driver_data, reinit_request);
-}
-EXPORT_SYMBOL_GPL(blk_mq_reinit_tagset);
-
 void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn,
 		void *priv)
 {
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 6bc29f73a9aa..cfd64e500d82 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -261,8 +261,6 @@ int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout);
 int blk_mq_tagset_iter(struct blk_mq_tag_set *set, void *data,
 		int (reinit_request)(void *, struct request *));
-int blk_mq_reinit_tagset(struct blk_mq_tag_set *set,
-			 int (reinit_request)(void *, struct request *));
 
 int blk_mq_map_queues(struct blk_mq_tag_set *set);
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues);
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 03:00:18 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:00:18 +0300
Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB
In-Reply-To: <20171001074203.GA11115@lst.de>
References: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com>
 <20171001074203.GA11115@lst.de>
Message-ID: <d2d574a7-3639-0917-3051-3ea4669de892@grimberg.me>

> ---
>  From b78f4164881125c4fecfdb87878d0120b2177c53 Mon Sep 17 00:00:00 2001
> From: Christoph Hellwig <hch at lst.de>
> Date: Sun, 1 Oct 2017 09:37:35 +0200
> Subject: nvme-pci: Use PCI bus address for data/queues in CMB
> 
> Currently, NVMe PCI host driver is programming CMB dma address as
> I/O SQs addresses. This results in failures on systems where 1:1
> outbound mapping is not used (example Broadcom iProc SOCs) because
> CMB BAR will be progammed with PCI bus address but NVMe PCI EP will
> try to access CMB using dma address.
> 
> To have CMB working on systems without 1:1 outbound mapping, we
> program PCI bus address for I/O SQs instead of dma address. This
> approach will work on systems with/without 1:1 outbound mapping.
> 
> Based on a report and previous patch from Abhishek Shah.
> 
> Fixes: 8ffaadf7 ("NVMe: Use CMB for the IO SQes if available")
> Cc: stable at vger.kernel.org
> Reported-by: Abhishek Shah <abhishek.shah at broadcom.com>
> Signed-off-by: Christoph Hellwig <hch at lst.de>

This looks good,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 03:01:06 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:01:06 +0300
Subject: [PATCH] nvme: simplify compat_ioctl handling
In-Reply-To: <20171005164646.17473-1-hch@lst.de>
References: <20171005164646.17473-1-hch@lst.de>
Message-ID: <8b075910-73a3-cf09-8369-13c4797f4b48@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 03:02:16 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:02:16 +0300
Subject: [PATCH v2 1/7] nvme core: allow controller RESETTING to
 RECONNECTING transition
In-Reply-To: <20171004082813.GA23467@infradead.org>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-2-jsmart2021@gmail.com>
 <20171004082813.GA23467@infradead.org>
Message-ID: <7b5f52d2-66ec-bdd8-de72-0bbb4cc2166b@grimberg.me>


>> Allow controller state transition : RESETTING to RECONNECTING
>>
>> Transport will typically transition from LIVE->RESETTING when
>> initially performing a reset or recovering from an error. Adding
>> this transition allows a transport to transition to RECONNECTING
>> when it checks/waits for connectivity then creates new transport
>> connections and reinits the controller.
> 
> Looks fine for now, although we really need to move to a common
> state machine..

Agree, will work on it,

> Reviewed-by: Christoph Hellwig <hch at lst.de>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 03:07:50 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:07:50 +0300
Subject: [PATCH v2 2/7] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <95dd2176-c5bb-386f-6a7a-4705576713fd@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-3-jsmart2021@gmail.com>
 <20171005075730.GA5063@infradead.org>
 <95dd2176-c5bb-386f-6a7a-4705576713fd@gmail.com>
Message-ID: <9b0d8baa-483d-3d76-19f8-30144cdc2aa8@grimberg.me>


>>> ????? changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
>>> -??? WARN_ON_ONCE(!changed);
>>> ????? ctrl->ctrl.nr_reconnects = 0;
>>> -??? nvme_start_ctrl(&ctrl->ctrl);
>>> +??? if (changed)
>>> +??????? nvme_start_ctrl(&ctrl->ctrl);
>>
>> It's just cosmetic, but can you folow the RDMA pattern here:
>>
>> ????changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
>> ????if (!changed) {
>> ??????? /* state change failure is ok if we're in DELETING state */
>> ??????? WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
>> ??????? return;
>> ????}
>>
>> else the patch looks fine:
>>
>> Reviewed-by: Christoph Hellwig <hch at lst.de>
>>
> 
> Actually, it's not just cosmetic. I don't want the warning to be 
> displayed.

That's wrong AFAICT, you do want a warning notification if you
failed to move the controller state to LIVE here and you are
_NOT_ in DELETING. This indicates a state transition that is
undefined.

> With FC's notification on device loss, in testing, we've seen 
> a device loss occur as this state transition occurs. I doubt that's 
> something rdma would ever see.

It does, hence the patch.

> With the device connectivity loss 
> notification, the state moves from NEW to RESETTING, then RECONNECTING.

you mean LIVE -> RESETTING -> RECONNECTING -> LIVE (on successful
reconnect).

That is exactly the point, the only way you fail RECONNECTING -> LIVE
is if you are in DELETING all of a sudden (triggered by the user
perhaps). Is there any way you will accept this state transition
to fail?


From sagi at grimberg.me  Wed Oct 11 03:11:02 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:11:02 +0300
Subject: [PATCH v2 3/7] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <20170927045046.22238-4-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-4-jsmart2021@gmail.com>
Message-ID: <984d8db1-c518-b515-05b5-46ba61c58c84@grimberg.me>


> +#define NVME_FC_DEFAULT_DEV_LOSS_TMO	60	/* seconds */

Why not reusing NVMF_DEF_CTRL_LOSS_TMO ? and isn't 60 seconds
a little to aggressive to give up?

What do you do with the ctrl_loss_tmo user parameter?


From sagi at grimberg.me  Wed Oct 11 03:14:09 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:14:09 +0300
Subject: [PATCH v2 4/7] nvme_fc: add dev_loss_tmo to controller
In-Reply-To: <20170927045046.22238-5-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-5-jsmart2021@gmail.com>
Message-ID: <b9a80a06-e5f2-c15f-1155-26c3fa071142@grimberg.me>

> +static void
> +nvme_fc_set_ctrl_devloss(struct nvme_fc_ctrl *ctrl,
> +		struct nvmf_ctrl_options *opts)
> +{
> +	u32 dev_loss_tmo;
> +
> +	/*
> +	 * dev_loss_tmo will be the max amount of time after an association
> +	 * failure that will be allowed for a new association to be
> +	 * established. It doesn't matter why the original association
> +	 * failed (FC connectivity loss, transport error, admin-request).
> +	 * The new association must be established before dev_loss_tmo
> +	 * expires or the controller will be torn down.
> +	 *
> +	 * If the connect parameters are less than the FC port dev_loss_tmo
> +	 * parameter, scale dev_loss_tmo to the connect parameters.
> +	 *
> +	 * If the connect parameters are larger than the FC port
> +	 * dev_loss_tmo parameter, adjust the connect parameters so that
> +	 * there is at least 1 attempt at a reconnect attempt before failing.
> +	 * Note: reconnects will be attempted only if there is FC connectivity.
> +	 */

OK now I see where you are accounting for fabrics ctrl_loss_tmo, but
given that we have a default, what is the use for your private FC port
default?


From sagi at grimberg.me  Wed Oct 11 03:17:23 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:17:23 +0300
Subject: [PATCH v2 4/7] nvme_fc: add dev_loss_tmo to controller
In-Reply-To: <9c9b8f0d-cc31-91ec-1be8-b5fd616dd0c5@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-5-jsmart2021@gmail.com>
 <20171005080040.GB5063@infradead.org>
 <9c9b8f0d-cc31-91ec-1be8-b5fd616dd0c5@gmail.com>
Message-ID: <81f8a5be-4c46-d603-6494-145f1032b692@grimberg.me>


> Hmm - this raise flags in my mind. I'm not sure how a dev_loss_tmo 
> applies to any other transport other than FC.

Can you explain what exactly do you mean by dev_loss_tmo only applies
to FC?


From sagi at grimberg.me  Wed Oct 11 03:18:20 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:18:20 +0300
Subject: [PATCH v2 6/7] nvme_fc: move remote port get/put/free location
In-Reply-To: <20170927045046.22238-7-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-7-jsmart2021@gmail.com>
Message-ID: <09565a4a-4214-a913-1359-cb1eed1a1bf3@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimerg.me>


From jthumshirn at suse.de  Wed Oct 11 03:28:25 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 12:28:25 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: stop controller reset if the
 controller is deleting
In-Reply-To: <1507715391-430-1-git-send-email-sagi@grimberg.me>
References: <1507715391-430-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171011102825.sisxrvt6jdgnn642@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 11 03:28:50 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 12:28:50 +0200
Subject: [PATCH v2 1/3] block: introduce blk_mq_tagset_iter
In-Reply-To: <1507715588-1041-2-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
 <1507715588-1041-2-git-send-email-sagi@grimberg.me>
Message-ID: <20171011102850.jfu4h6eizdqcza2n@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 11 03:29:33 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 12:29:33 +0200
Subject: [PATCH v2 3/3] block: remove blk_mq_reinit_tagset
In-Reply-To: <1507715588-1041-4-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
 <1507715588-1041-4-git-send-email-sagi@grimberg.me>
Message-ID: <20171011102933.su4duxtvk273reof@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Wed Oct 11 03:29:59 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:29:59 +0300
Subject: [PATCH v2 7/7] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20170927045046.22238-8-jsmart2021@gmail.com>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-8-jsmart2021@gmail.com>
Message-ID: <88124a13-30d4-8906-f9c7-6c00d6d7ee31@grimberg.me>


> This patch adds the dev_loss_tmo functionality to the transport.
> 
> When a remoteport is unregistered (connectivity lost), it is marked
> DELETED and the following is perfomed on all the controllers on the
> remoteport:
> - the controller is reset to delete the current association.
> - Once the association is terminated, the dev_loss_tmo timer is started.
>    A reconnect is not scheduled as there is no connectivity.
>    Note: the start of the dev_loss_tmo timer is in the generic
>    delete-association/create-new-association path. Thus it will be started
>    regardless of whether the reset was due to remote port connectivity
>    loss, a controller reset, or a transport run-time error.
> 
> When a remoteport is registered (connectivity established), the
> transport searches the list of remoteport structures that have pending
> deletions (controllers waiting to have dev_loss_tmo fire, thus
> preventing remoteport deletion). The transport looks for a matching
> wwnn/wwpn. If one is found, the remoteport is transitioned back to
> ONLINE, and the following occurs on all controllers on the remoteport:
> - any controllers in a RECONNECTING state have reconnection attempts
>    kicked off.
> - If the controller was RESETTING, it's natural RECONNECTING transition
>    will start a reconnect attempt.

OK, I think I finally understand the existence of the two tmos.
But may I ask, what happens if we only have ctrl_loss_tmo support?

If I understand correctly, it would behave exactly as expected as the
controller will periodically attempt reconnect until ctrl_loss_tmo fires
and then it is deleted. With dev_loss_tmo, once the lower between
ctrl_loss_tmo and dev_loss_tmo expires, the controller is deleted.
If my understanding is correct, this just adds a level of confusion
to setup procedure doesn't it?


From sagi at grimberg.me  Wed Oct 11 03:30:49 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 13:30:49 +0300
Subject: [PATCH v3] nvme_fc: fix iowait hang
In-Reply-To: <20171009203944.3541-1-jsmart2021@gmail.com>
References: <20171009203944.3541-1-jsmart2021@gmail.com>
Message-ID: <79ddd539-0244-cea3-f648-fad535540f42@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From jthumshirn at suse.de  Wed Oct 11 03:29:15 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 12:29:15 +0200
Subject: [PATCH v2 2/3] nvme: introduce nvme_reinit_tagset
In-Reply-To: <1507715588-1041-3-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
 <1507715588-1041-3-git-send-email-sagi@grimberg.me>
Message-ID: <20171011102915.ufeb54g474m33ydj@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Wed Oct 11 04:01:40 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:01:40 +0300
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <d06ea87b-1b0c-e581-b6f1-b4b2647eb431@grimberg.me>

Chaitanya,

Sorry for the late reply, was on vacation...

> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> 
> This adds SGL support for NVMe PCIe driver, based on an earlier patch
> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
> refactors the original code and adds new module parameter sgl_threshold
> to determine whether to use SGL or PRP for IOs.

One of the more appealing values of SGL support is removing the
virt_boundary constraint from the driver and use SGLs to issue
"gappy" IO. I would be very happy to see this happening, is this
something you plan to address?

This is why I wanted to not have a threshold based sgl/prp selection
as it does not allow us to remove the virt_boundary constraint for
controllers that don't have the "prp-able limitation".

We could have also a threshold to select the optimal selection for
sgl/prp, but it would need to meet the following conditions:
- choose prp if
    1. avg_seg_size is < sgl_threshold
    2. bio_vec is not gappy
(2) would mean traversing the bio_vec until we integrate this into
     the block layer. But given that this is cheap when the bio_vec
     is small and long bio_vecs its probably negligible compared to
     the longer transfer latency.
> +/*
> + * Calculates the number of pages needed for the SGL segments. For example a 4k
> + * page can accommodate 256 SGL descriptors.
> + */
> +static int nvme_npages_sgl(unsigned int num_seg)

Can you place nvme_pci_ prefix on all the new routines you add?

> +{
> +	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
> +}
> +
>   static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
> -		unsigned int size, unsigned int nseg)
> +		unsigned int size, unsigned int nseg, bool use_sgl)
>   {

I would suggest that also for functions that you touch their signature
make them nvme_pci_*.

> -	return sizeof(__le64 *) * nvme_npages(size, dev) +
> -			sizeof(struct scatterlist) * nseg;
> +	size_t alloc_size;
> +
> +	if (use_sgl)
> +		alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
> +	else
> +		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
> +
> +	return alloc_size + sizeof(struct scatterlist) * nseg;
>   }
>   
> -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>   {
>   	return sizeof(struct nvme_iod) +
> -		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
> +		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES, use_sgl);
>   }
>   
>   static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
>   	nvmeq->sq_tail = tail;
>   }
>   
> -static __le64 **iod_list(struct request *req)
> +static void **iod_list(struct request *req)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
> +	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>   }
>   
>   static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>   	unsigned int size = blk_rq_payload_bytes(rq);
>   
>   	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
> -		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
> +		size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
> +				iod->use_sgl);
> +
> +		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>   		if (!iod->sg)
>   			return BLK_STS_RESOURCE;
>   	} else {
> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
>   static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> -	const int last_prp = dev->ctrl.page_size / 8 - 1;
> +	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
> +	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>   	int i;
> -	__le64 **list = iod_list(req);
> -	dma_addr_t prp_dma = iod->first_dma;
>   
>   	if (iod->npages == 0)
> -		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
> +		dma_pool_free(dev->prp_small_pool, iod_list(req)[0], dma_addr);
> +
>   	for (i = 0; i < iod->npages; i++) {
> -		__le64 *prp_list = list[i];
> -		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
> -		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
> -		prp_dma = next_prp_dma;
> +		void *addr = iod_list(req)[i];
> +
> +		if (iod->use_sgl) {
> +			struct nvme_sgl_desc *sg_list = addr;
> +
> +			next_dma_addr =
> +				le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
> +		} else {
> +			__le64 *prp_list = addr;
> +
> +			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
> +		}
> +
> +		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
> +		dma_addr = next_dma_addr;
>   	}
>   
>   	if (iod->sg != iod->inline_sg)
> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
>   	}
>   }
>   
> -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmnd)
>   {
>   	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>   	struct dma_pool *pool;
> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   	u32 page_size = dev->ctrl.page_size;
>   	int offset = dma_addr & (page_size - 1);
>   	__le64 *prp_list;
> -	__le64 **list = iod_list(req);
> +	void **list = iod_list(req);
>   	dma_addr_t prp_dma;
>   	int nprps, i;
>   
> +	iod->use_sgl = false;
> +
>   	length -= (page_size - offset);
>   	if (length <= 0) {
>   		iod->first_dma = 0;
> -		return BLK_STS_OK;
> +		goto done;
>   	}
>   
>   	dma_len -= (page_size - offset);
> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   
>   	if (length <= page_size) {
>   		iod->first_dma = dma_addr;
> -		return BLK_STS_OK;
> +		goto done;
>   	}
>   
>   	nprps = DIV_ROUND_UP(length, page_size);
> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   		dma_len = sg_dma_len(sg);
>   	}
>   
> +done:
> +	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
> +	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
> +
>   	return BLK_STS_OK;
>   
>    bad_sgl:
> @@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
>   	return BLK_STS_IOERR;
>   }
>   
> +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
> +		struct scatterlist *sg)
> +{
> +	sge->addr = cpu_to_le64(sg_dma_address(sg));
> +	sge->length = cpu_to_le32(sg_dma_len(sg));
> +	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr, int entries)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
> +	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
> +}
> +
> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
> +		dma_addr_t dma_addr)
> +{
> +	sge->addr = cpu_to_le64(dma_addr);
> +	sge->length = cpu_to_le32(PAGE_SIZE);
> +	sge->type = NVME_SGL_FMT_SEG_DESC << 4;
> +}
> +
> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request *req,
> +		struct nvme_rw_command *cmd)
> +{

nvme_pci_*


From sagi at grimberg.me  Wed Oct 11 04:20:25 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:20:25 +0300
Subject: [PATCH 4/4] nvmet: Add feature close connection from target side
In-Reply-To: <1507635633-16665-5-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-5-git-send-email-israelr@mellanox.com>
Message-ID: <2b6780b5-29f6-72f3-5d40-9737710b94d9@grimberg.me>


> This allows the user to close any connection from target side
> by writing to the file "force_close" at the controller folder.
> Full path:
> /config/nvmet/subsystems/<SUBSYSTEM_NAME>/controllers/<CTRL_ID>/force_close

Why not rmdir?


From sagi at grimberg.me  Wed Oct 11 04:11:26 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:11:26 +0300
Subject: [PATCH 2/4] nvmet: Fix error flow in nvmet_alloc_ctrl()
In-Reply-To: <1507635633-16665-3-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-3-git-send-email-israelr@mellanox.com>
Message-ID: <3378b1a9-2dd8-0768-d691-51543d588a4c@grimberg.me>

Israel,

> Also rearrange nvmet_ctrl_free().

It would be better to separate into two patches.
In the nvmet_ctrl_free() patch, please explain
why is the rearrangement needed.


From sagi at grimberg.me  Wed Oct 11 04:09:40 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:09:40 +0300
Subject: [PATCH v2] nvme_fc: retry initial controller connections 3 times
In-Reply-To: <20171009233922.7216-1-jsmart2021@gmail.com>
References: <20171009233922.7216-1-jsmart2021@gmail.com>
Message-ID: <d80601a9-bead-2b21-fdaf-fea01fbdbafe@grimberg.me>


> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 1075488a81d6..18e036c6833b 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2790,7 +2790,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
>   {
>   	struct nvme_fc_ctrl *ctrl;
>   	unsigned long flags;
> -	int ret, idx;
> +	int ret, idx, retry;
>   
>   	if (!(rport->remoteport.port_role &
>   	    (FC_PORT_ROLE_NVME_DISCOVERY | FC_PORT_ROLE_NVME_TARGET))) {
> @@ -2882,9 +2882,37 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
>   	list_add_tail(&ctrl->ctrl_list, &rport->ctrl_list);
>   	spin_unlock_irqrestore(&rport->lock, flags);
>   
> -	ret = nvme_fc_create_association(ctrl);
> +	/*
> +	 * It's possible that transactions used to create the association
> +	 * may fail. Examples: CreateAssociation LS or CreateIOConnection
> +	 * LS gets dropped/corrupted/fails; or a frame gets dropped or a
> +	 * command times out for one of the actions to init the controller
> +	 * (Connect, Get/Set_Property, Set_Features, etc). Many of these
> +	 * transport errors (frame drop, LS failure) inherently must kill
> +	 * the association. The transport is coded so that any command used
> +	 * to create the association (prior to a LIVE state transition
> +	 * while NEW or RECONNECTING) will fail if it completes in error or
> +	 * times out.
> +	 *
> +	 * As such: as the connect request was mostly likely due to a
> +	 * udev event that discovered the remote port, meaning there is
> +	 * not an admin or script there to restart if the connect
> +	 * request fails, retry the initial connection creation up to
> +	 * three times before giving up and declaring failure.
> +	 */
> +	for (retry = 0; retry < 3; retry++) {
> +		ret = nvme_fc_create_association(ctrl);
> +		if (!ret)
> +			break;
> +	}
> +
>   	if (ret) {
> +		/* couldn't schedule retry - fail out */
> +		dev_err(ctrl->ctrl.device,
> +			"NVME-FC{%d}: Connect retry failed\n", ctrl->cnum);
> +
>   		ctrl->ctrl.opts = NULL;
> +
>   		/* initiate nvme ctrl ref counting teardown */
>   		nvme_uninit_ctrl(&ctrl->ctrl);
>   		nvme_put_ctrl(&ctrl->ctrl);
> 

Its not directly related to this patch, but the error handling here
seems non trivial, why isn't this going to the normal out_* error
targets below in the function?


From sagi at grimberg.me  Wed Oct 11 04:11:37 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:11:37 +0300
Subject: [PATCH 1/4] nvmet: Add port pointer to nvmet_ctrl structure
In-Reply-To: <1507635633-16665-2-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-2-git-send-email-israelr@mellanox.com>
Message-ID: <722fd123-d54f-c6fc-1e82-3ef4f37edb83@grimberg.me>

Looks fine,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 04:36:08 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:36:08 +0300
Subject: [PATCH 3/4] nvmet: Add controllers to configfs
In-Reply-To: <1507635633-16665-4-git-send-email-israelr@mellanox.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-4-git-send-email-israelr@mellanox.com>
Message-ID: <fc211d66-a762-8540-b2a9-9bc01a3a4c6a@grimberg.me>


> The commit show all the controllers and some info about them.
> This will allow the user to monitor the created controllers from target
> point of view.
> The "controllers" folder was added per subsystem under:
> /config/nvmet/subsystems/<SUBSYSTEM_NAME>/controllers/<CTRL_ID>/
> 
> <CTRL_ID> folder consists of:
>   - hostnqn: Host NQN
>   - port_traddr: Port Transport Address
>   - port_trsvcid: Port Transport Service ID
> 
> Signed-off-by: Israel Rukshin <israelr at mellanox.com>
> Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
> ---
>   drivers/nvme/target/configfs.c | 73 ++++++++++++++++++++++++++++++++++++++++++
>   drivers/nvme/target/core.c     |  6 ++++
>   drivers/nvme/target/nvmet.h    | 10 ++++++
>   3 files changed, 89 insertions(+)
> 
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index b6aeb1d..a0942c3 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -469,6 +469,49 @@ static struct config_group *nvmet_ns_make(struct config_group *group,
>   	.ct_owner		= THIS_MODULE,
>   };
>   
> +static ssize_t nvmet_ctrl_hostnqn_show(struct config_item *item, char *page)
> +{
> +	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
> +
> +	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->hostnqn);
> +}
> +
> +CONFIGFS_ATTR_RO(nvmet_ctrl_, hostnqn);
> +
> +static ssize_t nvmet_ctrl_traddr_show(struct config_item *item, char *page)
> +{
> +	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
> +
> +	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->port->disc_addr.traddr);
> +}
> +
> +CONFIGFS_ATTR_RO(nvmet_ctrl_, traddr);
> +
> +static ssize_t nvmet_ctrl_trsvcid_show(struct config_item *item, char *page)
> +{
> +	struct nvmet_ctrl *ctrl = to_nvmet_ctrl(item);
> +
> +	return snprintf(page, PAGE_SIZE, "%s\n", ctrl->port->disc_addr.trsvcid);
> +}
> +
> +CONFIGFS_ATTR_RO(nvmet_ctrl_, trsvcid);
> +
> +static struct configfs_attribute *nvmet_ctrl_attrs[] = {
> +	&nvmet_ctrl_attr_hostnqn,
> +	&nvmet_ctrl_attr_traddr,
> +	&nvmet_ctrl_attr_trsvcid,
> +	NULL,
> +};
> +
> +static struct config_item_type nvmet_ctrl_type = {
> +	.ct_attrs		= nvmet_ctrl_attrs,
> +	.ct_owner		= THIS_MODULE,
> +};
> +
> +static struct config_item_type nvmet_controllers_type = {
> +	.ct_owner		= THIS_MODULE,
> +};
> +
>   static int nvmet_port_subsys_allow_link(struct config_item *parent,
>   		struct config_item *target)
>   {
> @@ -760,6 +803,10 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
>   	configfs_add_default_group(&subsys->allowed_hosts_group,
>   			&subsys->group);
>   
> +	config_group_init_type_name(&subsys->controllers_group,
> +			"controllers", &nvmet_controllers_type);
> +	configfs_add_default_group(&subsys->controllers_group, &subsys->group);
> +

I think nvmet_ctrl is a much better fit as a config_item, perhaps we
need a kernel API to add a config_item? Or maybe it can bring all sorts
of races?

Christoph, what are your thoughts on this?


From sagi at grimberg.me  Wed Oct 11 04:39:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:39:42 +0300
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <20171010134844.hlgn7mif6yek2vvp@linux-x5ow.site>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
 <20171010134844.hlgn7mif6yek2vvp@linux-x5ow.site>
Message-ID: <bffb5035-3435-f66c-d4f3-cdc612a68e56@grimberg.me>


>> cp nvmefc_boot_connections.sh        /usr/sbin/lpfc/
> 
> [...]
> 
>> +ExecStart=/bin/sh -c "/usr/sbin/lpfc/nvmefc_boot_connections.sh"
> 
> /usr/sbin/lpfc wont get past our package reviewers, %_sbindir (for rpm, I'm
> sure there is an equivalent for deb as well) should be used there.
> 
> 
> Also wouldn't it make sense to have the scripts/services in nvme-cli?

It definitely would.


From sagi at grimberg.me  Wed Oct 11 04:40:05 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:40:05 +0300
Subject: [PATCH] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
References: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
Message-ID: <09911f37-7d22-5919-429b-a67767c192fb@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at rimberg.me>


From sagi at grimberg.me  Wed Oct 11 04:51:35 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:51:35 +0300
Subject: [PATCH 1/9] block: move REQ_NOWAIT
In-Reply-To: <20170925134031.10548-2-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-2-hch@lst.de>
Message-ID: <723f6977-f3e8-c9f6-b973-e1b62e321fa1@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 04:51:53 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:51:53 +0300
Subject: [PATCH 2/9] block: add REQ_DRV bit
In-Reply-To: <20170925134031.10548-3-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-3-hch@lst.de>
Message-ID: <a5db8f5c-98f9-3241-801b-b998a2f18b1d@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 04:59:25 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:59:25 +0300
Subject: [PATCH 6/9] nvme: introduce a nvme_ns_ids structure
In-Reply-To: <20170925134031.10548-7-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-7-hch@lst.de>
Message-ID: <f4a9295a-1834-3a17-6090-203e8507e3ef@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 11 05:04:14 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:04:14 +0300
Subject: [PATCH 7/9] nvme: track shared namespaces
In-Reply-To: <20170925134031.10548-8-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-8-hch@lst.de>
Message-ID: <68479c83-cdaa-77e0-232e-45c94b704765@grimberg.me>


> +/*
> + * Anchor structure for namespaces.  There is one for each namespace in a
> + * NVMe subsystem that any of our controllers can see, and the namespace
> + * structure for each controller is chained of it.  For private namespaces
> + * there is a 1:1 relation to our namespace structures, that is ->list
> + * only ever has a single entry for private namespaces.
> + */
> +struct nvme_ns_head {
> +	struct list_head	list;

Didn't we agree to call this list_head siblings?

> +	struct srcu_struct      srcu;
> +	unsigned		ns_id;
> +	struct nvme_ns_ids	ids;
> +	struct list_head	entry;
> +	struct kref		ref;
> +};
> +
>   struct nvme_ns {
>   	struct list_head list;
>   
>   	struct nvme_ctrl *ctrl;
>   	struct request_queue *queue;
>   	struct gendisk *disk;
> +	struct list_head siblings;

And this list_head list (or entry, or sibling, or sibling_entry)?

>   	struct nvm_dev *ndev;
>   	struct kref kref;
> +	struct nvme_ns_head *head;
>   	int instance;
>   
> -	unsigned ns_id;
> -	struct nvme_ns_ids ids;
>   	int lba_shift;
>   	u16 ms;
>   	u16 sgs;
> 


From sagi at grimberg.me  Wed Oct 11 04:54:47 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:54:47 +0300
Subject: [PATCH 3/9] block: provide a direct_make_request helper
In-Reply-To: <20170925134031.10548-4-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-4-hch@lst.de>
Message-ID: <eb2f78e6-042f-f39b-39e7-322c9a831dd2@grimberg.me>


> +/**
> + * direct_make_request - hand a buffer directly to its device driver for I/O
> + * @bio:  The bio describing the location in memory and on the device.
> + *
> + * This function behaves like generic_make_request(), but does not protect
> + * against recursion.  Must only be used if the called driver is known
> + * to not re-issue this bio or a split to itself.
> + */

Can you explain a bit on the comment? What happens if nvme attempts
to failover a split if the original IO didn't comply to the
virt_boundary?

Would that cause any problems?


From sagi at grimberg.me  Wed Oct 11 04:57:31 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 14:57:31 +0300
Subject: [PATCH 5/9] nvme: track subsystems
In-Reply-To: <20170925134031.10548-6-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-6-hch@lst.de>
Message-ID: <1535c6f9-8ddf-ab30-2bbd-5d9ce626b913@grimberg.me>

> @@ -2230,7 +2315,7 @@ static ssize_t nvme_sysfs_show_subsysnqn(struct device *dev,
>   {
>   	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>   
> -	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subnqn);
> +	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subsys->subnqn);
>   }
>   static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
>   
> @@ -2770,12 +2855,22 @@ EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
>   static void nvme_free_ctrl(struct kref *kref)
>   {
>   	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
> +	struct nvme_subsystem *subsys = ctrl->subsys;
>   
>   	put_device(ctrl->device);
>   	nvme_release_instance(ctrl);
>   	ida_destroy(&ctrl->ns_ida);
>   
> +	if (subsys) {
> +		mutex_lock(&subsys->lock);
> +		list_del(&ctrl->subsys_entry);
> +		mutex_unlock(&subsys->lock);
> +	}
> +

When can this happen? can a controller not have a subsys?


From sagi at grimberg.me  Wed Oct 11 05:24:50 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:24:50 +0300
Subject: [PATCH 8/9] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20170925134031.10548-9-hch@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-9-hch@lst.de>
Message-ID: <725347b9-661a-1c95-02d1-3c10b57dd8d4@grimberg.me>


> +static const struct block_device_operations nvme_subsys_ops = {
> +	.owner		= THIS_MODULE,
> +};

Nit - maybe better to name this nvme_ns_head_ops

> +	ret = -ENOMEM;
> +	q = blk_alloc_queue_node(GFP_KERNEL, NUMA_NO_NODE);
> +	if (!q)
> +		goto out_free_head;
> +	q->queuedata = head;
> +	blk_queue_make_request(q, nvme_make_request);
> +	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
> +	/* set to a default value for 512 until disk is validated */
> +	blk_queue_logical_block_size(q, 512);
> +	nvme_set_queue_limits(ctrl, q);
> +
> +	head->disk = alloc_disk(0);
> +	if (!head->disk)
> +		goto out_cleanup_queue;
> +	head->disk->fops = &nvme_subsys_ops;
> +	head->disk->private_data = head;
> +	head->disk->queue = q;
> +	head->disk->flags = GENHD_FL_EXT_DEVT;
> +	sprintf(head->disk->disk_name, "nvm-sub%dn%d",
> +			ctrl->subsys->instance, nsid);

Did we end up in agreement on nvmsXnY?


> @@ -2669,8 +2905,10 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>   	}
>   
>   	mutex_lock(&ns->ctrl->subsys->lock);
> -	if (head)
> +	if (head) {
> +		rcu_assign_pointer(head->current_path, NULL);

Doesn't this assignment need to happen only when the ns *is* the
current_path?


From sagi at grimberg.me  Wed Oct 11 05:29:05 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:05 +0300
Subject: [PATCH 0/7] Some nvme-rdma host patches for 4.15
Message-ID: <1507724952-13917-1-git-send-email-sagi@grimberg.me>

A couple of minor fixes. Patch 6 is improving the reconnect
flow to not teardown the admin/io queues in every reconnect
(which doesn't really happen as we check a state on the queues).

Sagi Grimberg (7):
  nvme-rdma: pass tagset to directly nvme_rdma_free_tagset
  nvme-rdma: fix wrong logging message
  nvme-rdma: move assignment to declaration
  nvme-rdma: Check that reinit_request got a proper mr
  nvme-rdma: Don't local invalidate if the queue is not live
  nvme-rdma: teardown admin/io queues once on error recovery
  nvme-rdma: change queue flag semantics DELETING -> ALLOCATED

 drivers/nvme/host/rdma.c | 61 +++++++++++++++++++++++-------------------------
 1 file changed, 29 insertions(+), 32 deletions(-)

-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:06 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:06 +0300
Subject: [PATCH 1/7] nvme-rdma: pass tagset to directly nvme_rdma_free_tagset
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-2-git-send-email-sagi@grimberg.me>

Instead of flagging admin/io.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index cf2ac5d893d5..9e040bbc08c9 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -670,11 +670,10 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl)
 	return ret;
 }
 
-static void nvme_rdma_free_tagset(struct nvme_ctrl *nctrl, bool admin)
+static void nvme_rdma_free_tagset(struct nvme_ctrl *nctrl,
+		struct blk_mq_tag_set *set)
 {
 	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
-	struct blk_mq_tag_set *set = admin ?
-			&ctrl->admin_tag_set : &ctrl->tag_set;
 
 	blk_mq_free_tag_set(set);
 	nvme_rdma_dev_put(ctrl->device);
@@ -744,7 +743,7 @@ static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	nvme_rdma_stop_queue(&ctrl->queues[0]);
 	if (remove) {
 		blk_cleanup_queue(ctrl->ctrl.admin_q);
-		nvme_rdma_free_tagset(&ctrl->ctrl, true);
+		nvme_rdma_free_tagset(&ctrl->ctrl, ctrl->ctrl.admin_tagset);
 	}
 	nvme_rdma_free_queue(&ctrl->queues[0]);
 }
@@ -818,7 +817,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 		blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_free_tagset:
 	if (new)
-		nvme_rdma_free_tagset(&ctrl->ctrl, true);
+		nvme_rdma_free_tagset(&ctrl->ctrl, ctrl->ctrl.admin_tagset);
 out_free_queue:
 	nvme_rdma_free_queue(&ctrl->queues[0]);
 	return error;
@@ -830,7 +829,7 @@ static void nvme_rdma_destroy_io_queues(struct nvme_rdma_ctrl *ctrl,
 	nvme_rdma_stop_io_queues(ctrl);
 	if (remove) {
 		blk_cleanup_queue(ctrl->ctrl.connect_q);
-		nvme_rdma_free_tagset(&ctrl->ctrl, false);
+		nvme_rdma_free_tagset(&ctrl->ctrl, ctrl->ctrl.tagset);
 	}
 	nvme_rdma_free_io_queues(ctrl);
 }
@@ -873,7 +872,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 		blk_cleanup_queue(ctrl->ctrl.connect_q);
 out_free_tag_set:
 	if (new)
-		nvme_rdma_free_tagset(&ctrl->ctrl, false);
+		nvme_rdma_free_tagset(&ctrl->ctrl, ctrl->ctrl.tagset);
 out_free_io_queues:
 	nvme_rdma_free_io_queues(ctrl);
 	return ret;
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:07 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:07 +0300
Subject: [PATCH 2/7] nvme-rdma: fix wrong logging message
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-3-git-send-email-sagi@grimberg.me>

Not necessarily address resolution failed.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 9e040bbc08c9..09f774abe5a8 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -544,7 +544,7 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
 	ret = nvme_rdma_wait_for_cm(queue);
 	if (ret) {
 		dev_info(ctrl->ctrl.device,
-			"rdma_resolve_addr wait failed (%d).\n", ret);
+			"rdma connection establishment failed (%d)\n", ret);
 		goto out_destroy_cm_id;
 	}
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:08 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:08 +0300
Subject: [PATCH 3/7] nvme-rdma: move assignment to declaration
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-4-git-send-email-sagi@grimberg.me>

No need for the extra line for trivial assignments.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 09f774abe5a8..6e62657662fd 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -434,11 +434,9 @@ nvme_rdma_find_get_device(struct rdma_cm_id *cm_id)
 
 static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue)
 {
-	struct nvme_rdma_device *dev;
-	struct ib_device *ibdev;
+	struct nvme_rdma_device *dev = queue->device;
+	struct ib_device *ibdev = dev->dev;
 
-	dev = queue->device;
-	ibdev = dev->dev;
 	rdma_destroy_qp(queue->cm_id);
 	ib_free_cq(queue->ib_cq);
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:09 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:09 +0300
Subject: [PATCH 4/7] nvme-rdma: Check that reinit_request got a proper mr
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-5-git-send-email-sagi@grimberg.me>

Warn if req->mr is NULL as it should never happen.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 6e62657662fd..93f4dd0b6f26 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -274,6 +274,9 @@ static int nvme_rdma_reinit_request(void *data, struct request *rq)
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
 	int ret = 0;
 
+	if (WARN_ON_ONCE(!req->mr))
+		return 0;
+
 	ib_dereg_mr(req->mr);
 
 	req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG,
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:10 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:10 +0300
Subject: [PATCH 5/7] nvme-rdma: Don't local invalidate if the queue is not live
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-6-git-send-email-sagi@grimberg.me>

No chance for the local invalidate to succeed if the queue-pair
is in error state. Most likely the target will do a remote
invalidation of our mr so not a big loss on the test_bit.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 93f4dd0b6f26..e3fe57011fcb 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1055,7 +1055,7 @@ static void nvme_rdma_unmap_data(struct nvme_rdma_queue *queue,
 	if (!blk_rq_bytes(rq))
 		return;
 
-	if (req->mr->need_inval) {
+	if (req->mr->need_inval && test_bit(NVME_RDMA_Q_LIVE, &req->queue->flags)) {
 		res = nvme_rdma_inv_rkey(queue, req);
 		if (unlikely(res < 0)) {
 			dev_err(ctrl->ctrl.device,
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:11 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:11 +0300
Subject: [PATCH 6/7] nvme-rdma: teardown admin/io queues once on error recovery
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-7-git-send-email-sagi@grimberg.me>

Relying on the queue state while tearing down on every reconnect
attempt is not a good design. We should do it once in err_work
and simply try to establish the queues for each reconnect attempt.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index e3fe57011fcb..2a18465bf361 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -925,10 +925,6 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
 
 	++ctrl->ctrl.nr_reconnects;
 
-	if (ctrl->ctrl.queue_count > 1)
-		nvme_rdma_destroy_io_queues(ctrl, false);
-
-	nvme_rdma_destroy_admin_queue(ctrl, false);
 	ret = nvme_rdma_configure_admin_queue(ctrl, false);
 	if (ret)
 		goto requeue;
@@ -936,7 +932,7 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
 	if (ctrl->ctrl.queue_count > 1) {
 		ret = nvme_rdma_configure_io_queues(ctrl, false);
 		if (ret)
-			goto requeue;
+			goto destroy_admin;
 	}
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
@@ -946,14 +942,17 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	ctrl->ctrl.nr_reconnects = 0;
-
 	nvme_start_ctrl(&ctrl->ctrl);
 
-	dev_info(ctrl->ctrl.device, "Successfully reconnected\n");
+	dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attepmpt)\n",
+			ctrl->ctrl.nr_reconnects);
+
+	ctrl->ctrl.nr_reconnects = 0;
 
 	return;
 
+destroy_admin:
+	nvme_rdma_destroy_admin_queue(ctrl, false);
 requeue:
 	dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
 			ctrl->ctrl.nr_reconnects);
@@ -969,17 +968,15 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
 	if (ctrl->ctrl.queue_count > 1) {
 		nvme_stop_queues(&ctrl->ctrl);
-		nvme_rdma_stop_io_queues(ctrl);
-	}
-	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
-	nvme_rdma_stop_queue(&ctrl->queues[0]);
-
-	/* We must take care of fastfail/requeue all our inflight requests */
-	if (ctrl->ctrl.queue_count > 1)
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,
 					nvme_cancel_request, &ctrl->ctrl);
+		nvme_rdma_destroy_io_queues(ctrl, false);
+	}
+
+	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_cancel_request, &ctrl->ctrl);
+	nvme_rdma_destroy_admin_queue(ctrl, false);
 
 	/*
 	 * queues are not a live anymore, so restart the queues to fail fast
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 11 05:29:12 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 11 Oct 2017 15:29:12 +0300
Subject: [PATCH 7/7] nvme-rdma: change queue flag semantics DELETING ->
 ALLOCATED
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <1507724952-13917-8-git-send-email-sagi@grimberg.me>

Instead of marking we are deleting, mark we
are allocated and check that instead. This makes
the logic symmetrical to connected mark check.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 2a18465bf361..405895b1dff2 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -79,8 +79,8 @@ struct nvme_rdma_request {
 };
 
 enum nvme_rdma_queue_flags {
-	NVME_RDMA_Q_LIVE		= 0,
-	NVME_RDMA_Q_DELETING		= 1,
+	NVME_RDMA_Q_ALLOCATED		= 0,
+	NVME_RDMA_Q_LIVE		= 1,
 };
 
 struct nvme_rdma_queue {
@@ -549,7 +549,7 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
 		goto out_destroy_cm_id;
 	}
 
-	clear_bit(NVME_RDMA_Q_DELETING, &queue->flags);
+	set_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags);
 
 	return 0;
 
@@ -569,7 +569,7 @@ static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
 
 static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
 {
-	if (test_and_set_bit(NVME_RDMA_Q_DELETING, &queue->flags))
+	if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
 		return;
 
 	nvme_rdma_destroy_queue_ib(queue);
-- 
2.7.4



From jthumshirn at suse.de  Wed Oct 11 06:33:20 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 15:33:20 +0200
Subject: [PATCH 1/7] nvme-rdma: pass tagset to directly
 nvme_rdma_free_tagset
In-Reply-To: <1507724952-13917-2-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-2-git-send-email-sagi@grimberg.me>
Message-ID: <20171011133320.j2uj4hco5gpomeqv@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 11 06:34:10 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 15:34:10 +0200
Subject: [PATCH 2/7] nvme-rdma: fix wrong logging message
In-Reply-To: <1507724952-13917-3-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-3-git-send-email-sagi@grimberg.me>
Message-ID: <20171011133410.wglrx4ozjlfk2dds@linux-x5ow.site>

heh yeah that one's puzzled me quite a bit the 1st time I tried nvme-rdma,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 11 06:34:35 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 15:34:35 +0200
Subject: [PATCH 3/7] nvme-rdma: move assignment to declaration
In-Reply-To: <1507724952-13917-4-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-4-git-send-email-sagi@grimberg.me>
Message-ID: <20171011133435.lgselq25zwjbjsbu@linux-x5ow.site>

Fair enough,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 11 06:35:27 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 11 Oct 2017 15:35:27 +0200
Subject: [PATCH 4/7] nvme-rdma: Check that reinit_request got a proper mr
In-Reply-To: <1507724952-13917-5-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-5-git-send-email-sagi@grimberg.me>
Message-ID: <20171011133527.ajkxucplizu4753r@linux-x5ow.site>

Out of curiosity, did this happen to you as a motivation for the patch?

Anyways,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From james.smart at broadcom.com  Wed Oct 11 07:29:36 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 11 Oct 2017 07:29:36 -0700
Subject: [PATCH v2 2/7] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <9b0d8baa-483d-3d76-19f8-30144cdc2aa8@grimberg.me>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-3-jsmart2021@gmail.com>
 <20171005075730.GA5063@infradead.org>
 <95dd2176-c5bb-386f-6a7a-4705576713fd@gmail.com>
 <9b0d8baa-483d-3d76-19f8-30144cdc2aa8@grimberg.me>
Message-ID: <22a8ac57-3087-1f16-2624-10c0fe86fb88@broadcom.com>

On 10/11/2017 3:07 AM, Sagi Grimberg wrote:
>
>>>> ????? changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
>>>> -??? WARN_ON_ONCE(!changed);
>>>> ????? ctrl->ctrl.nr_reconnects = 0;
>>>> -??? nvme_start_ctrl(&ctrl->ctrl);
>>>> +??? if (changed)
>>>> +??????? nvme_start_ctrl(&ctrl->ctrl);
>>>
>>> It's just cosmetic, but can you folow the RDMA pattern here:
>>>
>>> ????changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
>>> ????if (!changed) {
>>> ??????? /* state change failure is ok if we're in DELETING state */
>>> ??????? WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
>>> ??????? return;
>>> ????}
>>>
>>> else the patch looks fine:
>>>
>>> Reviewed-by: Christoph Hellwig <hch at lst.de>
>>>
>>
>> Actually, it's not just cosmetic. I don't want the warning to be 
>> displayed.
>
> That's wrong AFAICT, you do want a warning notification if you
> failed to move the controller state to LIVE here and you are
> _NOT_ in DELETING. This indicates a state transition that is
> undefined.

No.? The WARN_ON is a once only - and puts out a dastardly set of 
messages with stack trace. Users see this as a bad error the way its 
formatted.

Instead - the messages that are displayed when the other states are 
transitioned to (already present) give you all the notification you need 
and in a manner that is easy to understand.

"state transition that is not defined" - it is a defined transition, 
otherwise the state machine wouldn't have allowed it.


>
>> With the device connectivity loss notification, the state moves from 
>> NEW to RESETTING, then RECONNECTING.
>
> you mean LIVE -> RESETTING -> RECONNECTING -> LIVE (on successful
> reconnect).

yes

>
> That is exactly the point, the only way you fail RECONNECTING -> LIVE
> is if you are in DELETING all of a sudden (triggered by the user
> perhaps). 

true - or triggered by a FC device connectivity loss for an extended period.

> Is there any way you will accept this state transition
> to fail?

?? which transaction. FC follows the state transitions per the state 
machine.

-- james





From guanjx09 at 163.com  Wed Oct 11 07:32:40 2017
From: guanjx09 at 163.com (Guan Junxiong)
Date: Wed, 11 Oct 2017 22:32:40 +0800
Subject: [PATCH 2/6] nvme: use 'nvmsXnZ' instead of 'nvm-subXnZ'
In-Reply-To: <20171002162038.GS8463@localhost.localdomain>
References: <1506952559-1588-1-git-send-email-hare@suse.de>
 <1506952559-1588-3-git-send-email-hare@suse.de>
 <20171002161621.GB11048@lst.de> <20171002162038.GS8463@localhost.localdomain>
Message-ID: <d9796b73-5309-a5e5-679b-dd0c67437e2b@163.com>



On Tuesday, October 03, 2017 12:20 AM, Keith Busch wrote:
> On Mon, Oct 02, 2017 at 06:16:21PM +0200, Christoph Hellwig wrote:
>> On Mon, Oct 02, 2017 at 03:55:55PM +0200, Hannes Reinecke wrote:
>>> Align with overall nvme device naming scheme.
>>
>> It is very close to the normal /dev/nvme* devices, so I feat it will
>> create a lot of confusion.
>>
>> But given that all device name suggestions so far suck I'd be happy
>> to take a stawpoll of linux-nvme subscribers :)
> 
> Of the options availble, +1 to Hannes' nvmsXnY.
> 

+1 for Hanne's nvmsXnY

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
> 



From james.smart at broadcom.com  Wed Oct 11 07:35:08 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 11 Oct 2017 07:35:08 -0700
Subject: [PATCH v2 3/7] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <984d8db1-c518-b515-05b5-46ba61c58c84@grimberg.me>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-4-jsmart2021@gmail.com>
 <984d8db1-c518-b515-05b5-46ba61c58c84@grimberg.me>
Message-ID: <d79a9eca-5fbc-1933-986e-bb16467e34cc@broadcom.com>

On 10/11/2017 3:11 AM, Sagi Grimberg wrote:
>
>> +#define NVME_FC_DEFAULT_DEV_LOSS_TMO 60??? /* seconds */
>
> Why not reusing NVMF_DEF_CTRL_LOSS_TMO ? and isn't 60 seconds
> a little to aggressive to give up?

Because there is an existing FC port timeout which is currently over on 
the SCSI side of the house.? The FC port will support both SCSI and 
FC-NVME simultaneously on the FC port - so it spans/affects both.? When 
the "common fc class" is created and moved out from under scsi, it will 
be the one with the value.


>
> What do you do with the ctrl_loss_tmo user parameter?

Well - this patch munged the dev_loss_tmo value into the 
ctrl_loss_tmo-based values. That is an issue - so in a prior comment I 
stated I'm reposting with the controller loss timeout kept separate from 
the fc port dev_loss_tmo. Thus NVMF_DEF_CTRL_LOSS_TMO and ctrl_loss_tmo 
will be used normally - like it is with RDMA. Dev_loss_tmo will be 
tracked at the fc port level and if fires, will trump wherever the 
controller is in its ctrl_loss_tmo

-- james





From maxg at mellanox.com  Wed Oct 11 07:35:22 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Wed, 11 Oct 2017 17:35:22 +0300
Subject: [PATCH 4/4] nvmet: Add feature close connection from target side
In-Reply-To: <2b6780b5-29f6-72f3-5d40-9737710b94d9@grimberg.me>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-5-git-send-email-israelr@mellanox.com>
 <2b6780b5-29f6-72f3-5d40-9737710b94d9@grimberg.me>
Message-ID: <7b7c1c96-dee7-d5ee-6ffe-10cb125dc1f2@mellanox.com>



On 10/11/2017 2:20 PM, Sagi Grimberg wrote:
> 
>> This allows the user to close any connection from target side
>> by writing to the file "force_close" at the controller folder.
>> Full path:
>> /config/nvmet/subsystems/<SUBSYSTEM_NAME>/controllers/<CTRL_ID>/force_close 
>>
> 
> Why not rmdir?

Theese dirs weren't created by mkdir..


From james.smart at broadcom.com  Wed Oct 11 07:52:13 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 11 Oct 2017 07:52:13 -0700
Subject: [PATCH v2 7/7] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <88124a13-30d4-8906-f9c7-6c00d6d7ee31@grimberg.me>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-8-jsmart2021@gmail.com>
 <88124a13-30d4-8906-f9c7-6c00d6d7ee31@grimberg.me>
Message-ID: <68b78559-e75a-7c2d-a588-c6063954d16a@broadcom.com>

On 10/11/2017 3:29 AM, Sagi Grimberg wrote:
> OK, I think I finally understand the existence of the two tmos.
> But may I ask, what happens if we only have ctrl_loss_tmo support?
>
> If I understand correctly, it would behave exactly as expected as the
> controller will periodically attempt reconnect until ctrl_loss_tmo fires
> and then it is deleted. With dev_loss_tmo, once the lower between
> ctrl_loss_tmo and dev_loss_tmo expires, the controller is deleted.
> If my understanding is correct, this just adds a level of confusion
> to setup procedure doesn't it?

agree with your statements. And no, it doesn't add confusion. You are 
ignoring the whole history coming from the SCSI side that is expected 
and well-known. If anything, adding the per-controller ctrl_loss_tmo 
that has different values, making fc-nvme behave differently than scsi 
on the same port (running scsi and fc concurrently perhaps), is what 
adds confusion.

-- james




From james.smart at broadcom.com  Wed Oct 11 08:07:36 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 11 Oct 2017 08:07:36 -0700
Subject: [PATCH v2] nvme_fc: retry initial controller connections 3 times
In-Reply-To: <d80601a9-bead-2b21-fdaf-fea01fbdbafe@grimberg.me>
References: <20171009233922.7216-1-jsmart2021@gmail.com>
 <d80601a9-bead-2b21-fdaf-fea01fbdbafe@grimberg.me>
Message-ID: <542d4c8e-54fc-0fd7-c29d-5cf1ed2934fa@broadcom.com>

On 10/11/2017 4:09 AM, Sagi Grimberg wrote:
> Its not directly related to this patch, but the error handling here
> seems non trivial, why isn't this going to the normal out_* error
> targets below in the function?

Because, while debugging the original flow which did teardown in the 
calling sequence like rdma, had a lot of concurrency issues and didn't 
fit well with a more generic create_association/delete_association back 
end. The issue is how many things are real hw contexts in FC that have 
to be flushed with the lldd before delete can really occur.? Those paths 
are all solved already for the LIVE->DELETING flows using ref-counting.? 
So, I perform the calling-sequence teardown until the nvme controller is 
init'd, at which point I defer to the normal refcounting delete flows. 
There's a comment that highlights that.

-- james



From james.smart at broadcom.com  Wed Oct 11 08:10:51 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 11 Oct 2017 08:10:51 -0700
Subject: [PATCH v2 4/7] nvme_fc: add dev_loss_tmo to controller
In-Reply-To: <81f8a5be-4c46-d603-6494-145f1032b692@grimberg.me>
References: <20170927045046.22238-1-jsmart2021@gmail.com>
 <20170927045046.22238-5-jsmart2021@gmail.com>
 <20171005080040.GB5063@infradead.org>
 <9c9b8f0d-cc31-91ec-1be8-b5fd616dd0c5@gmail.com>
 <81f8a5be-4c46-d603-6494-145f1032b692@grimberg.me>
Message-ID: <f483c5dc-e5c3-ab1d-4419-919b8515dbb1@broadcom.com>

On 10/11/2017 3:17 AM, Sagi Grimberg wrote:
>
>> Hmm - this raise flags in my mind. I'm not sure how a dev_loss_tmo 
>> applies to any other transport other than FC.
>
> Can you explain what exactly do you mean by dev_loss_tmo only applies
> to FC?

Well I'm at a loss at how dev_loss_tmo could apply to another transport 
like RDMA or TCP. I'd like someone to explain it to me.

afaik - no other transport has events from the fabric when connectivity 
changes (perhaps on the otherside of a switch) thus the transport 
actively tracks connectivity to the other endpoint.? FC does. So we know 
almost immediately when a target port goes away. Dev_loss_tmo is a 
duration after initial connectivity loss where we wait and suspend 
devices attached to the fc port - both scsi and fc-nvme - allowing the 
port to come back and the os-device to resume operation. if it expires, 
the device is deleted.? It's existed on SCSI for many years.? So it's 
basically the same behavior as the ctrl_loss_tmo, but the timer is at 
the port level and spans multiple ctrl's. Given they were so similar, I 
munged the ctrl_loss_tmo and dev_loss_tmo together and timed things at 
the controller only. Now I'm regretting that choice and keeping things 
in their natural place: ctlr_loss_tmo a the ctlr level (regardless of 
connectivity, with immediate reconnect reschedule if no connectivity); 
and dev_loss_tmo at the fc port level (connectivity loss starts 
reset/reconnect on all, timeout deletes all).

-- james



From maxg at mellanox.com  Wed Oct 11 11:38:19 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Wed, 11 Oct 2017 21:38:19 +0300
Subject: [PATCH 1/7] nvme-rdma: pass tagset to directly
 nvme_rdma_free_tagset
In-Reply-To: <1507724952-13917-2-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-2-git-send-email-sagi@grimberg.me>
Message-ID: <91983fd0-074d-3600-e043-08a9c41110b7@mellanox.com>



On 10/11/2017 3:29 PM, Sagi Grimberg wrote:
> Instead of flagging admin/io.
> 
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   drivers/nvme/host/rdma.c | 13 ++++++-------
>   1 file changed, 6 insertions(+), 7 deletions(-)
> 

Looks good,
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From maxg at mellanox.com  Wed Oct 11 11:40:02 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Wed, 11 Oct 2017 21:40:02 +0300
Subject: [PATCH 2/7] nvme-rdma: fix wrong logging message
In-Reply-To: <1507724952-13917-3-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-3-git-send-email-sagi@grimberg.me>
Message-ID: <432db184-8ed8-cc98-60fd-a2dbeab71a70@mellanox.com>



On 10/11/2017 3:29 PM, Sagi Grimberg wrote:
> Not necessarily address resolution failed.
> 
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   drivers/nvme/host/rdma.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 

Looks good,
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From maxg at mellanox.com  Wed Oct 11 11:44:33 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Wed, 11 Oct 2017 21:44:33 +0300
Subject: [PATCH 3/7] nvme-rdma: move assignment to declaration
In-Reply-To: <1507724952-13917-4-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-4-git-send-email-sagi@grimberg.me>
Message-ID: <4f596c70-f13f-7dd5-641e-285569ff505e@mellanox.com>



On 10/11/2017 3:29 PM, Sagi Grimberg wrote:
> No need for the extra line for trivial assignments.
> 
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

Looks fine,
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From yizhan at redhat.com  Thu Oct 12 01:24:32 2017
From: yizhan at redhat.com (Yi Zhang)
Date: Thu, 12 Oct 2017 16:24:32 +0800
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
Message-ID: <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>


> Lets take this one step at a time, starting with this issue.
>
> First, there is a reason why a simple create_ctrl fails, can we isolate
> exactly which call fails? Was something else going on that might have
> made the simple create_ctrl fail?
>
> We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
> messages but we do see "creating I/O queues" which means that we either
> failed at IO tagset allocation or initializing connect_q.
>
> We have a missing error code assignment so can you try the following 
> patch:

Hi Sagi
Sorry for the late response, here is the log with your patch, let me 
know if you need more test.

[  236.980239] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  237.131416] nvme_rdma: nvme_rdma_free_tagset: freeing ADMIN tagset
[  237.173614] nvme nvme0: creating 40 I/O queues.
[  237.688843] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  243.384596] nvme nvme0: rescanning
[  243.527922] nvme nvme0: Reconnecting in 10 seconds...
[  254.010547] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  254.018052] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  254.024482] nvme nvme0: Failed reconnect attempt 1
[  254.029844] nvme nvme0: Reconnecting in 10 seconds...
[  264.238223] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  264.245727] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  264.252159] nvme nvme0: Failed reconnect attempt 2
[  264.257545] nvme nvme0: Reconnecting in 10 seconds...
[  274.477977] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  274.485470] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  274.491905] nvme nvme0: Failed reconnect attempt 3
[  274.497287] nvme nvme0: Reconnecting in 10 seconds...
[  284.737691] nvme nvme0: creating 40 I/O queues.
[  284.745163] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 1011432003)
[  284.754617] ------------[ cut here ]------------
[  284.759790] WARNING: CPU: 12 PID: 6 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  284.770010] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  284.849181]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  284.870886] CPU: 12 PID: 6 Comm: kworker/u368:0 Not tainted 
4.14.0-rc4.10.02.sagi+ #30
[  284.879746] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  284.888133] Workqueue: ib_addr process_one_req [ib_core]
[  284.894074] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  284.900699] RIP: 0010:__domain_mapping+0x367/0x380
[  284.906051] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202
[  284.911899] RAX: 0000000000000004 RBX: 0000001011432003 RCX: 
0000000000000000
[  284.920643] RDX: 0000000000000000 RSI: ffff88103e78e058 RDI: 
ffff88103e78e058
[  284.929371] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 
0000000000000000
[  284.938088] R10: 00000000000002f7 R11: 0000000001011432 R12: 
ffff88201ab634e8
[  284.946814] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  284.955533] FS:  0000000000000000(0000) GS:ffff88103e780000(0000) 
knlGS:0000000000000000
[  284.965318] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  284.972491] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 
00000000001606e0
[  284.981227] Call Trace:
[  284.984712]  __intel_map_single+0xeb/0x180
[  284.990059]  intel_alloc_coherent+0xb5/0x130
[  284.995584]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
[  285.001690]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
[  285.008754]  mlx4_ib_create_cq+0x223/0x450 [mlx4_ib]
[  285.015027]  ib_alloc_cq+0x49/0x170 [ib_core]
[  285.020634]  nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma]
[  285.027416]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
[  285.033689]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.039417]  process_one_req+0x8d/0x120 [ib_core]
[  285.045408]  process_one_work+0x149/0x360
[  285.050620]  worker_thread+0x4d/0x3c0
[  285.055432]  kthread+0x109/0x140
[  285.059780]  ? rescuer_thread+0x380/0x380
[  285.064987]  ? kthread_park+0x60/0x60
[  285.069805]  ret_from_fork+0x25/0x30
[  285.074511] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.097088] ---[ end trace 501470fecce57311 ]---
[  285.103299] DMAR: DRHD: handling fault status reg 2
[  285.103761] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 100526b002)
[  285.103766] ------------[ cut here ]------------
[  285.103768] WARNING: CPU: 12 PID: 6 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  285.103768] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  285.103785]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  285.103791] CPU: 12 PID: 6 Comm: kworker/u368:0 Tainted: G W       
4.14.0-rc4.10.02.sagi+ #30
[  285.103792] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  285.103797] Workqueue: ib_addr process_one_req [ib_core]
[  285.103798] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  285.103799] RIP: 0010:__domain_mapping+0x367/0x380
[  285.103800] RSP: 0018:ffffc9000009fc48 EFLAGS: 00010206
[  285.103800] RAX: 0000000000000003 RBX: 000000100526b002 RCX: 
0000000000000000
[  285.103801] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 
0000000000000282
[  285.103801] RBP: ffffc9000009fca8 R08: 0000000000000050 R09: 
0000000000000000
[  285.103802] R10: 0000000000000860 R11: 000000000100526b R12: 
ffff88201ab634e8
[  285.103803] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  285.103803] FS:  0000000000000000(0000) GS:ffff88103e780000(0000) 
knlGS:0000000000000000
[  285.103804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.103805] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 
00000000001606e0
[  285.103805] Call Trace:
[  285.103808]  __intel_map_single+0xeb/0x180
[  285.103809]  intel_map_page+0x39/0x40
[  285.103811]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  285.103813]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.103817]  process_one_req+0x8d/0x120 [ib_core]
[  285.103818]  process_one_work+0x149/0x360
[  285.103819]  worker_thread+0x4d/0x3c0
[  285.103821]  kthread+0x109/0x140
[  285.103822]  ? rescuer_thread+0x380/0x380
[  285.103823]  ? kthread_park+0x60/0x60
[  285.103825]  ret_from_fork+0x25/0x30
[  285.103826] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.103839] ---[ end trace 501470fecce57312 ]---
[  285.190580] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 100d51b001)
[  285.190586] ------------[ cut here ]------------
[  285.190590] WARNING: CPU: 30 PID: 757 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  285.190590] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  285.190609]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  285.190616] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  285.190617] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  285.190619] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  285.190620] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  285.190622] RIP: 0010:__domain_mapping+0x367/0x380
[  285.190622] RSP: 0018:ffffc9000851fc60 EFLAGS: 00010202
[  285.190623] RAX: 0000000000000002 RBX: 000000100d51b001 RCX: 
0000000000000000
[  285.190624] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 
0000000000000292
[  285.190624] RBP: ffffc9000851fcc0 R08: 0000000000000050 R09: 
0000000000000000
[  285.190625] R10: 0000000000000881 R11: 000000000100d51b R12: 
ffff88201ab634e8
[  285.190625] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  285.190626] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  285.190626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.190627] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  285.190628] Call Trace:
[  285.190630]  __intel_map_single+0xeb/0x180
[  285.190632]  intel_map_page+0x39/0x40
[  285.190636]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
[  285.190642]  ib_alloc_mr+0x26/0x50 [ib_core]
[  285.190644]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
[  285.190645]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
[  285.190647]  blk_mq_reinit_tagset+0x5c/0x90
[  285.190649]  nvme_rdma_configure_io_queues+0x204/0x280 [nvme_rdma]
[  285.190650]  nvme_rdma_reconnect_ctrl_work+0x5b/0xe0 [nvme_rdma]
[  285.190652]  process_one_work+0x149/0x360
[  285.190653]  worker_thread+0x4d/0x3c0
[  285.190655]  kthread+0x109/0x140
[  285.190656]  ? rescuer_thread+0x380/0x380
[  285.190657]  ? kthread_park+0x60/0x60
[  285.190660]  ret_from_fork+0x25/0x30
[  285.190660] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.190673] ---[ end trace 501470fecce57313 ]---
[  285.819126] DMAR: [DMA Write] Request device [04:00.0] fault addr 
e0e9d000 [fault reason 05] PTE Write access is not set
[  286.035086] nvme nvme0: Successfully reconnected
[  286.035143] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with 
status memory management operation error (6)
[  286.187508] nvme nvme0: Reconnecting in 10 seconds...
[  296.270043] nvme nvme0: creating 40 I/O queues.
[  296.336505] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
1012082002 not 1012082002)
[  296.346704] ------------[ cut here ]------------
[  296.352618] WARNING: CPU: 32 PID: 815 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  296.363755] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  296.447983]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  296.471170] CPU: 32 PID: 815 Comm: kworker/u368:1 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  296.482344] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  296.491493] Workqueue: ib_addr process_one_req [ib_core]
[  296.498221] task: ffff8810373d2e80 task.stack: ffffc9000849c000
[  296.505613] RIP: 0010:__domain_mapping+0x367/0x380
[  296.511751] RSP: 0018:ffffc9000849fc48 EFLAGS: 00010202
[  296.518380] RAX: 0000000000000001 RBX: 0000001012082002 RCX: 
0000000000000000
[  296.527146] RDX: 0000000000000000 RSI: ffff88103ea0e058 RDI: 
ffff88103ea0e058
[  296.535918] RBP: ffffc9000849fca8 R08: 0000000000000000 R09: 
0000000000000000
[  296.544691] R10: 00000000000003ff R11: 0000000001012082 R12: 
ffff88201ab634e8
[  296.553462] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  296.562240] FS:  0000000000000000(0000) GS:ffff88103ea00000(0000) 
knlGS:0000000000000000
[  296.572078] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  296.579292] CR2: 00007fdc7ae90000 CR3: 0000001034df0002 CR4: 
00000000001606e0
[  296.588070] Call Trace:
[  296.591620]  __intel_map_single+0xeb/0x180
[  296.597012]  intel_map_page+0x39/0x40
[  296.601912]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  296.608760]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  296.614540]  process_one_req+0x8d/0x120 [ib_core]
[  296.620602]  process_one_work+0x149/0x360
[  296.625883]  worker_thread+0x4d/0x3c0
[  296.630779]  kthread+0x109/0x140
[  296.635177]  ? rescuer_thread+0x380/0x380
[  296.640453]  ? kthread_park+0x60/0x60
[  296.645341]  ret_from_fork+0x25/0x30
[  296.650132] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  296.672876] ---[ end trace 501470fecce57314 ]---
[  297.231475] nvme nvme0: Successfully reconnected
[  297.231523] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with 
status memory management operation error (6)
[  297.378026] nvme nvme0: Reconnecting in 10 seconds...
[  307.765849] ------------[ cut here ]------------
[  307.771793] kernel BUG at drivers/iommu/iova.c:836!
[  307.778007] invalid opcode: 0000 [#1] SMP
[  307.783257] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  307.867936]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  307.891229] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  307.902427] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  307.911579] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  307.919847] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  307.927232] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  307.933827] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[  307.940415] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 
0000000100400038
[  307.949132] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: 
ffff8810372c4e58
[  307.957857] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 
0000000100400038
[  307.966586] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: 
ffff880ffda45000
[  307.975308] R13: ffff8810372c4e58 R14: 0000000000000046 R15: 
ffff880ffda45000
[  307.984032] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  307.993834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  308.001012] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  308.009749] Call Trace:
[  308.013243]  free_iova_fast+0xc9/0x1d0
[  308.018198]  fq_ring_free+0x78/0xc0
[  308.022845]  queue_iova+0x111/0x130
[  308.027478]  intel_unmap+0xa5/0x130
[  308.032118]  intel_unmap_page+0xe/0x10
[  308.037044]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.044789]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.051951]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.058529]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.065786]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.073240]  process_one_work+0x149/0x360
[  308.078470]  worker_thread+0x4d/0x3c0
[  308.083299]  kthread+0x109/0x140
[  308.087652]  ? rescuer_thread+0x380/0x380
[  308.092878]  ? kthread_park+0x60/0x60
[  308.097699]  ret_from_fork+0x25/0x30
[  308.102407] Code: ef e8 ef fe ff ff 48 63 d3 49 3b 14 24 72 d7 4c 89 
f6 4c 89 ef e8 3b 3d 2b 00 49 c7 04 24 00 00 00 00 5b 41 5c 41 5d 41 5e 
5d c3 <0f> 0b 0f 1f 84 00 00 00 00 00 0
[  308.124980] RIP: iova_magazine_free_pfns+0x76/0x80 RSP: ffffc9000851fc58
[  308.133265] ---[ end trace 501470fecce57315 ]---
[  308.142811] Kernel panic - not syncing: Fatal exception
[  308.149425] Kernel Offset: disabled
[  308.158679] ---[ end Kernel panic - not syncing: Fatal exception
[  308.166135] ------------[ cut here ]------------
[  308.172010] WARNING: CPU: 30 PID: 757 at kernel/sched/core.c:1179 
set_task_cpu+0x191/0x1a0
[  308.181959] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  308.266001]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  308.289112] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G D 
W       4.14.0-rc4.10.02.sagi+ #30
[  308.300236] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  308.309330] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  308.317552] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  308.324905] RIP: 0010:set_task_cpu+0x191/0x1a0
[  308.330621] RSP: 0018:ffff88103e9c3c38 EFLAGS: 00010046
[  308.337191] RAX: 0000000000000200 RBX: ffff881034cc5d00 RCX: 
0000000000000001
[  308.345896] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 
ffff881034cc5d00
[  308.354604] RBP: ffff88103e9c3c58 R08: 0000000000000000 R09: 
0000000000000000
[  308.363314] R10: 00000000000003ff R11: 0000000000000001 R12: 
ffff881034cc68ac
[  308.372018] R13: 0000000000000000 R14: 0000000000000000 R15: 
000000000001b900
[  308.380720] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  308.390495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  308.397632] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  308.406347] Call Trace:
[  308.409809]  <IRQ>
[  308.412789]  try_to_wake_up+0x166/0x470
[  308.417816]  default_wake_function+0x12/0x20
[  308.423325]  __wake_up_common+0x8a/0x160
[  308.428438]  __wake_up_locked+0x16/0x20
[  308.433450]  ep_poll_callback+0xd0/0x300
[  308.438562]  __wake_up_common+0x8a/0x160
[  308.443664]  __wake_up_common_lock+0x7e/0xc0
[  308.449150]  __wake_up+0x13/0x20
[  308.453455]  wake_up_klogd_work_func+0x40/0x60
[  308.459124]  irq_work_run_list+0x4d/0x70
[  308.464217]  ? tick_sched_do_timer+0x70/0x70
[  308.469701]  irq_work_tick+0x40/0x50
[  308.474409]  update_process_times+0x42/0x60
[  308.479800]  tick_sched_handle+0x2d/0x60
[  308.484881]  tick_sched_timer+0x39/0x70
[  308.489852]  __hrtimer_run_queues+0xe5/0x230
[  308.495291]  hrtimer_interrupt+0xa8/0x1a0
[  308.500466]  smp_apic_timer_interrupt+0x5f/0x130
[  308.506320]  apic_timer_interrupt+0x9d/0xb0
[  308.511661]  </IRQ>
[  308.514658] RIP: 0010:panic+0x1fd/0x245
[  308.519603] RSP: 0018:ffffc9000851f9a8 EFLAGS: 00000246 ORIG_RAX: 
ffffffffffffff10
[  308.528740] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 
0000000000000006
[  308.537352] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 
ffff88103e9ce050
[  308.545943] RBP: ffffc9000851fa18 R08: 0000000000000000 R09: 
00000000000008f9
[  308.554511] R10: 00000000000003ff R11: 0000000000000001 R12: 
ffffffff81a3f340
[  308.563067] R13: 0000000000000000 R14: 0000000000000000 R15: 
0000000000000004
[  308.571619]  ? panic+0x1f6/0x245
[  308.575773]  oops_end+0xb8/0xd0
[  308.579822]  die+0x42/0x50
[  308.583348]  do_trap+0x16a/0x180
[  308.587442]  do_error_trap+0x89/0x110
[  308.592007]  ? iova_magazine_free_pfns+0x76/0x80
[  308.597624]  ? rem_res_range+0x1ae/0x200 [mlx4_core]
[  308.603625]  do_invalid_op+0x20/0x30
[  308.608063]  invalid_op+0x18/0x20
[  308.612198] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  308.618473] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[  308.624735] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 
0000000100400038
[  308.633146] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: 
ffff8810372c4e58
[  308.641555] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 
0000000100400038
[  308.649969] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: 
ffff880ffda45000
[  308.658387] R13: ffff8810372c4e58 R14: 0000000000000046 R15: 
ffff880ffda45000
[  308.666799]  ? iova_magazine_free_pfns+0x3e/0x80
[  308.672399]  free_iova_fast+0xc9/0x1d0
[  308.677046]  fq_ring_free+0x78/0xc0
[  308.681370]  queue_iova+0x111/0x130
[  308.685691]  intel_unmap+0xa5/0x130
[  308.690011]  intel_unmap_page+0xe/0x10
[  308.694626]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.702056]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.708909]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.715182]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.722148]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.729311]  process_one_work+0x149/0x360
[  308.734242]  worker_thread+0x4d/0x3c0
[  308.738783]  kthread+0x109/0x140
[  308.742840]  ? rescuer_thread+0x380/0x380
[  308.747774]  ? kthread_park+0x60/0x60
[  308.752309]  ret_from_fork+0x25/0x30
[  308.756747] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf 
fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe 
ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 0
[  308.778800] ---[ end trace 501470fecce57316 ]---


From bhumirks at gmail.com  Thu Oct 12 05:38:21 2017
From: bhumirks at gmail.com (Bhumika Goyal)
Date: Thu, 12 Oct 2017 14:38:21 +0200
Subject: [PATCH] nvmet: make config_item_type const
Message-ID: <1507811901-11522-1-git-send-email-bhumirks@gmail.com>

This is a followup patch for:
https://patchwork.kernel.org/patch/9999649/ and
https://lkml.org/lkml/2017/10/11/375

Make config_item_type structures const as they are either passed to a
function having the argument as const or used inside an if statement or
stored in the const "ci_type" field of a config_item structure.

Done using Coccinelle

Signed-off-by: Bhumika Goyal <bhumirks at gmail.com>
---
 drivers/nvme/target/configfs.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index b6aeb1d..e6b2d2a 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -20,8 +20,8 @@
 
 #include "nvmet.h"
 
-static struct config_item_type nvmet_host_type;
-static struct config_item_type nvmet_subsys_type;
+static const struct config_item_type nvmet_host_type;
+static const struct config_item_type nvmet_subsys_type;
 
 /*
  * nvmet_port Generic ConfigFS definitions.
@@ -425,7 +425,7 @@ static void nvmet_ns_release(struct config_item *item)
 	.release		= nvmet_ns_release,
 };
 
-static struct config_item_type nvmet_ns_type = {
+static const struct config_item_type nvmet_ns_type = {
 	.ct_item_ops		= &nvmet_ns_item_ops,
 	.ct_attrs		= nvmet_ns_attrs,
 	.ct_owner		= THIS_MODULE,
@@ -464,7 +464,7 @@ static struct config_group *nvmet_ns_make(struct config_group *group,
 	.make_group		= nvmet_ns_make,
 };
 
-static struct config_item_type nvmet_namespaces_type = {
+static const struct config_item_type nvmet_namespaces_type = {
 	.ct_group_ops		= &nvmet_namespaces_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -540,7 +540,7 @@ static void nvmet_port_subsys_drop_link(struct config_item *parent,
 	.drop_link		= nvmet_port_subsys_drop_link,
 };
 
-static struct config_item_type nvmet_port_subsys_type = {
+static const struct config_item_type nvmet_port_subsys_type = {
 	.ct_item_ops		= &nvmet_port_subsys_item_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -613,7 +613,7 @@ static void nvmet_allowed_hosts_drop_link(struct config_item *parent,
 	.drop_link		= nvmet_allowed_hosts_drop_link,
 };
 
-static struct config_item_type nvmet_allowed_hosts_type = {
+static const struct config_item_type nvmet_allowed_hosts_type = {
 	.ct_item_ops		= &nvmet_allowed_hosts_item_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -729,7 +729,7 @@ static void nvmet_subsys_release(struct config_item *item)
 	.release		= nvmet_subsys_release,
 };
 
-static struct config_item_type nvmet_subsys_type = {
+static const struct config_item_type nvmet_subsys_type = {
 	.ct_item_ops		= &nvmet_subsys_item_ops,
 	.ct_attrs		= nvmet_subsys_attrs,
 	.ct_owner		= THIS_MODULE,
@@ -767,7 +767,7 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	.make_group		= nvmet_subsys_make,
 };
 
-static struct config_item_type nvmet_subsystems_type = {
+static const struct config_item_type nvmet_subsystems_type = {
 	.ct_group_ops		= &nvmet_subsystems_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -827,7 +827,7 @@ static void nvmet_referral_release(struct config_item *item)
 	.release	= nvmet_referral_release,
 };
 
-static struct config_item_type nvmet_referral_type = {
+static const struct config_item_type nvmet_referral_type = {
 	.ct_owner	= THIS_MODULE,
 	.ct_attrs	= nvmet_referral_attrs,
 	.ct_item_ops	= &nvmet_referral_item_ops,
@@ -852,7 +852,7 @@ static struct config_group *nvmet_referral_make(
 	.make_group		= nvmet_referral_make,
 };
 
-static struct config_item_type nvmet_referrals_type = {
+static const struct config_item_type nvmet_referrals_type = {
 	.ct_owner	= THIS_MODULE,
 	.ct_group_ops	= &nvmet_referral_group_ops,
 };
@@ -880,7 +880,7 @@ static void nvmet_port_release(struct config_item *item)
 	.release		= nvmet_port_release,
 };
 
-static struct config_item_type nvmet_port_type = {
+static const struct config_item_type nvmet_port_type = {
 	.ct_attrs		= nvmet_port_attrs,
 	.ct_item_ops		= &nvmet_port_item_ops,
 	.ct_owner		= THIS_MODULE,
@@ -921,7 +921,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 	.make_group		= nvmet_ports_make,
 };
 
-static struct config_item_type nvmet_ports_type = {
+static const struct config_item_type nvmet_ports_type = {
 	.ct_group_ops		= &nvmet_ports_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -940,7 +940,7 @@ static void nvmet_host_release(struct config_item *item)
 	.release		= nvmet_host_release,
 };
 
-static struct config_item_type nvmet_host_type = {
+static const struct config_item_type nvmet_host_type = {
 	.ct_item_ops		= &nvmet_host_item_ops,
 	.ct_owner		= THIS_MODULE,
 };
@@ -963,14 +963,14 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
 	.make_group		= nvmet_hosts_make_group,
 };
 
-static struct config_item_type nvmet_hosts_type = {
+static const struct config_item_type nvmet_hosts_type = {
 	.ct_group_ops		= &nvmet_hosts_group_ops,
 	.ct_owner		= THIS_MODULE,
 };
 
 static struct config_group nvmet_hosts_group;
 
-static struct config_item_type nvmet_root_type = {
+static const struct config_item_type nvmet_root_type = {
 	.ct_owner		= THIS_MODULE,
 };
 
-- 
1.9.1



From bhumirks at gmail.com  Thu Oct 12 08:03:50 2017
From: bhumirks at gmail.com (Bhumika Goyal)
Date: Thu, 12 Oct 2017 17:03:50 +0200
Subject: [PATCH] nvmet: make config_item_type const
In-Reply-To: <1507811901-11522-1-git-send-email-bhumirks@gmail.com>
References: <1507811901-11522-1-git-send-email-bhumirks@gmail.com>
Message-ID: <CAOH+1jFc0ZRrKks7f2R1NXWYmebmCfF8D2Bwv_ajEzi=10m0JA@mail.gmail.com>

On Thu, Oct 12, 2017 at 2:38 PM, Bhumika Goyal <bhumirks at gmail.com> wrote:
> This is a followup patch for:
> https://patchwork.kernel.org/patch/9999649/ and
> https://lkml.org/lkml/2017/10/11/375
>
> Make config_item_type structures const as they are either passed to a
> function having the argument as const or used inside an if statement or
> stored in the const "ci_type" field of a config_item structure.
>
> Done using Coccinelle
>

Actually, this patch is dependent on the patches in the links
https://lkml.org/lkml/2017/10/11/375 and
https://patchwork.kernel.org/patch/9999649/. Therefore, this patch
won't be correct unless the patches in these links gets applied.

> Signed-off-by: Bhumika Goyal <bhumirks at gmail.com>
> ---
>  drivers/nvme/target/configfs.c | 30 +++++++++++++++---------------
>  1 file changed, 15 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index b6aeb1d..e6b2d2a 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -20,8 +20,8 @@
>
>  #include "nvmet.h"
>
> -static struct config_item_type nvmet_host_type;
> -static struct config_item_type nvmet_subsys_type;
> +static const struct config_item_type nvmet_host_type;
> +static const struct config_item_type nvmet_subsys_type;
>
>  /*
>   * nvmet_port Generic ConfigFS definitions.
> @@ -425,7 +425,7 @@ static void nvmet_ns_release(struct config_item *item)
>         .release                = nvmet_ns_release,
>  };
>
> -static struct config_item_type nvmet_ns_type = {
> +static const struct config_item_type nvmet_ns_type = {
>         .ct_item_ops            = &nvmet_ns_item_ops,
>         .ct_attrs               = nvmet_ns_attrs,
>         .ct_owner               = THIS_MODULE,
> @@ -464,7 +464,7 @@ static struct config_group *nvmet_ns_make(struct config_group *group,
>         .make_group             = nvmet_ns_make,
>  };
>
> -static struct config_item_type nvmet_namespaces_type = {
> +static const struct config_item_type nvmet_namespaces_type = {
>         .ct_group_ops           = &nvmet_namespaces_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -540,7 +540,7 @@ static void nvmet_port_subsys_drop_link(struct config_item *parent,
>         .drop_link              = nvmet_port_subsys_drop_link,
>  };
>
> -static struct config_item_type nvmet_port_subsys_type = {
> +static const struct config_item_type nvmet_port_subsys_type = {
>         .ct_item_ops            = &nvmet_port_subsys_item_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -613,7 +613,7 @@ static void nvmet_allowed_hosts_drop_link(struct config_item *parent,
>         .drop_link              = nvmet_allowed_hosts_drop_link,
>  };
>
> -static struct config_item_type nvmet_allowed_hosts_type = {
> +static const struct config_item_type nvmet_allowed_hosts_type = {
>         .ct_item_ops            = &nvmet_allowed_hosts_item_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -729,7 +729,7 @@ static void nvmet_subsys_release(struct config_item *item)
>         .release                = nvmet_subsys_release,
>  };
>
> -static struct config_item_type nvmet_subsys_type = {
> +static const struct config_item_type nvmet_subsys_type = {
>         .ct_item_ops            = &nvmet_subsys_item_ops,
>         .ct_attrs               = nvmet_subsys_attrs,
>         .ct_owner               = THIS_MODULE,
> @@ -767,7 +767,7 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
>         .make_group             = nvmet_subsys_make,
>  };
>
> -static struct config_item_type nvmet_subsystems_type = {
> +static const struct config_item_type nvmet_subsystems_type = {
>         .ct_group_ops           = &nvmet_subsystems_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -827,7 +827,7 @@ static void nvmet_referral_release(struct config_item *item)
>         .release        = nvmet_referral_release,
>  };
>
> -static struct config_item_type nvmet_referral_type = {
> +static const struct config_item_type nvmet_referral_type = {
>         .ct_owner       = THIS_MODULE,
>         .ct_attrs       = nvmet_referral_attrs,
>         .ct_item_ops    = &nvmet_referral_item_ops,
> @@ -852,7 +852,7 @@ static struct config_group *nvmet_referral_make(
>         .make_group             = nvmet_referral_make,
>  };
>
> -static struct config_item_type nvmet_referrals_type = {
> +static const struct config_item_type nvmet_referrals_type = {
>         .ct_owner       = THIS_MODULE,
>         .ct_group_ops   = &nvmet_referral_group_ops,
>  };
> @@ -880,7 +880,7 @@ static void nvmet_port_release(struct config_item *item)
>         .release                = nvmet_port_release,
>  };
>
> -static struct config_item_type nvmet_port_type = {
> +static const struct config_item_type nvmet_port_type = {
>         .ct_attrs               = nvmet_port_attrs,
>         .ct_item_ops            = &nvmet_port_item_ops,
>         .ct_owner               = THIS_MODULE,
> @@ -921,7 +921,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
>         .make_group             = nvmet_ports_make,
>  };
>
> -static struct config_item_type nvmet_ports_type = {
> +static const struct config_item_type nvmet_ports_type = {
>         .ct_group_ops           = &nvmet_ports_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -940,7 +940,7 @@ static void nvmet_host_release(struct config_item *item)
>         .release                = nvmet_host_release,
>  };
>
> -static struct config_item_type nvmet_host_type = {
> +static const struct config_item_type nvmet_host_type = {
>         .ct_item_ops            = &nvmet_host_item_ops,
>         .ct_owner               = THIS_MODULE,
>  };
> @@ -963,14 +963,14 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
>         .make_group             = nvmet_hosts_make_group,
>  };
>
> -static struct config_item_type nvmet_hosts_type = {
> +static const struct config_item_type nvmet_hosts_type = {
>         .ct_group_ops           = &nvmet_hosts_group_ops,
>         .ct_owner               = THIS_MODULE,
>  };
>
>  static struct config_group nvmet_hosts_group;
>
> -static struct config_item_type nvmet_root_type = {
> +static const struct config_item_type nvmet_root_type = {
>         .ct_owner               = THIS_MODULE,
>  };
>
> --
> 1.9.1
>


From maxg at mellanox.com  Thu Oct 12 08:58:46 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Thu, 12 Oct 2017 18:58:46 +0300
Subject: [PATCH 3/4] nvmet: Add controllers to configfs
In-Reply-To: <d06594ae-225e-8065-bead-019c99f309ad@gmail.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-4-git-send-email-israelr@mellanox.com>
 <d06594ae-225e-8065-bead-019c99f309ad@gmail.com>
Message-ID: <f8a2c50d-2f4e-588a-e39d-3c4692037bb5@mellanox.com>



On 10/12/2017 3:16 PM, roy wrote:
> hi,
> 
> Consider exposing all the info about the port (trtype, adrfam, etc') 
> maybe by symlink the controller to to port config_item.

We tought about it, but since we create the dirs dynamically (and not by 
the user performing mkdir), we prefered not to hack this symlink at 
first stage.


From axboe at kernel.dk  Thu Oct 12 15:52:51 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Thu, 12 Oct 2017 16:52:51 -0600
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <20171012224550.25440-2-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
Message-ID: <ee12eeaf-9848-478d-4ebe-9d3497e248e0@kernel.dk>

On 10/12/2017 04:45 PM, Bart Van Assche wrote:
> Many kernel drivers contain code that allocate and free both a
> scatterlist and the pages that populate that scatterlist.
> Introduce functions under lib/ that perform these tasks instead
> of duplicating this functionality in multiple drivers.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> ---
>  include/linux/sgl_alloc.h |  16 ++++++++
>  lib/Kconfig               |   4 ++
>  lib/Makefile              |   1 +
>  lib/sgl_alloc.c           | 102 ++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 123 insertions(+)
>  create mode 100644 include/linux/sgl_alloc.h
>  create mode 100644 lib/sgl_alloc.c
> 
> diff --git a/include/linux/sgl_alloc.h b/include/linux/sgl_alloc.h
> new file mode 100644
> index 000000000000..a0f719690c9e
> --- /dev/null
> +++ b/include/linux/sgl_alloc.h
> @@ -0,0 +1,16 @@
> +#ifndef _SGL_ALLOC_H_
> +#define _SGL_ALLOC_H_
> +
> +#include <linux/types.h> /* bool, gfp_t */
> +
> +struct scatterlist;
> +
> +struct scatterlist *sgl_alloc_order(unsigned long long length,
> +				    unsigned int order, unsigned int *nent_p,
> +				    gfp_t gfp, bool chainable);
> +struct scatterlist *sgl_alloc(unsigned long long length, unsigned int *nent_p,
> +			      gfp_t gfp);
> +void sgl_free_order(struct scatterlist *sgl, int order);
> +void sgl_free(struct scatterlist *sgl);
> +
> +#endif /* _SGL_ALLOC_H_ */

Should this just go into linux/scatterlist.h instead of creating a new header
file?

-- 
Jens Axboe



From Bart.VanAssche at wdc.com  Thu Oct 12 16:00:10 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Thu, 12 Oct 2017 23:00:10 +0000
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <ee12eeaf-9848-478d-4ebe-9d3497e248e0@kernel.dk>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
 <ee12eeaf-9848-478d-4ebe-9d3497e248e0@kernel.dk>
Message-ID: <1507849209.14112.1.camel@wdc.com>

On Thu, 2017-10-12 at 16:52 -0600, Jens Axboe wrote:
> On 10/12/2017 04:45 PM, Bart Van Assche wrote:
> > +++ b/include/linux/sgl_alloc.h
> > @@ -0,0 +1,16 @@
> > +#ifndef _SGL_ALLOC_H_
> > +#define _SGL_ALLOC_H_
> > +
> > +#include <linux/types.h> /* bool, gfp_t */
> > +
> > +struct scatterlist;
> > +
> > +struct scatterlist *sgl_alloc_order(unsigned long long length,
> > +				    unsigned int order, unsigned int *nent_p,
> > +				    gfp_t gfp, bool chainable);
> > +struct scatterlist *sgl_alloc(unsigned long long length, unsigned int *nent_p,
> > +			      gfp_t gfp);
> > +void sgl_free_order(struct scatterlist *sgl, int order);
> > +void sgl_free(struct scatterlist *sgl);
> > +
> > +#endif /* _SGL_ALLOC_H_ */
> 
> Should this just go into linux/scatterlist.h instead of creating a new header
> file?

That's something I can change. I don't have a strong opinion about whether
these declarations should go into a new header file or into linux/scatterlist.h.

Bart.

From ckulkarnilinux at gmail.com  Thu Oct 12 20:19:27 2017
From: ckulkarnilinux at gmail.com (chaitany kulkarni)
Date: Thu, 12 Oct 2017 20:19:27 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
 <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
Message-ID: <CAK0tqikZJCOzYoQSi4E1DS41RdnVrt477GunFajO0dF58Y=aTA@mail.gmail.com>

On Sun, Oct 8, 2017 at 2:16 AM, Max Gurtovoy <maxg at mellanox.com> wrote:
>
>
> On 10/5/2017 10:48 PM, Chaitanya Kulkarni wrote:
>>
>> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>>
>> This adds SGL support for NVMe PCIe driver, based on an earlier patch
>> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
>> refactors the original code and adds new module parameter sgl_threshold
>> to determine whether to use SGL or PRP for IOs.
>>
>> The usage of SGLs is controlled by the sgl_threshold module parameter,
>> which allows to conditionally use SGLs if average request segment size
>> (avg_seg_size) is greater than sgl_threshold. In the original patch,
>> the decision of using SGLs was dependent only on the IO size,
>> with the new approach we consider not only IO size but also the
>> number of physical segments present in the IO.
>>
>> We calculate avg_seg_size based on request payload bytes and number
>> of physical segments present in the request.
>
>
> Can you add some performance numbers here to show the benefit ?
> and also please recomend the user when it's better to use sgls vs prps.
>
[CK] Yes, once we finish with all the code changes I'll collect the numbers
on the final version.
>>
>> For e.g.:-
>>
>> 1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
>> avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.
>>
>> 2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
>> avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.
>>
>> 3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
>> avg_seg_size = 4K use sgl if avg_seg_size
>
>
> typo :)
[CK] I'll fix it in the next version.
>
>>
>> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>> ---
>>   drivers/nvme/host/pci.c | 222
>> ++++++++++++++++++++++++++++++++++++++++++------
>>   1 file changed, 197 insertions(+), 25 deletions(-)
>>
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index cb73bc8..56db876 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -45,6 +45,8 @@
>>    */
>>   #define NVME_AQ_BLKMQ_DEPTH   (NVME_AQ_DEPTH - NVME_NR_AERS)
>>   +#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct
>> nvme_sgl_desc))
>
>
> should we use PAGE_SIZE or ctrl->page_size ?
[CK]In nvme_setup_sgls() there are two cases where we allocate the SGL.
Both the cases are dependant on the way we setup the DMA pools.

In function nvme_setup_prp_pools() it uses 256 for dev->prp_small_pool &
PAGE_SIZE for dev->prp_page_pool as a pool block size parameter.

When we setup the SGLs at that time we allocate memory from either of these
pools. For the first case (entries <= (256 / sizeof(struct nvme_sgl_desc)))
it uses small pool, for the second case it uses dev->prp_page_pool.
As mentioned above dev->prp_page_pool uses PAGE_SIZE for size of the
blocks for which we allocate the memory. So in order to calculate the
number of entries which are going in the allocated DMA block (sgl_list)
we need to use PAGE_SIZE.
>
>> +
>>   static int use_threaded_interrupts;
>>   module_param(use_threaded_interrupts, int, 0);
>>   @@ -57,6 +59,10 @@
>>   MODULE_PARM_DESC(max_host_mem_size_mb,
>>         "Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
>>   +static unsigned int sgl_threshold = SZ_32K;
>> +module_param(sgl_threshold, uint, 0644);
>> +MODULE_PARM_DESC(sgl_threshold, "use SGL for I/O larger or equal to this
>> size");
>> +
>
>
> please mention that 0 value disable sgl usage.
[CK] Sounds good, I'll add this in the next version of this patch.
>
>
>>   static int io_queue_depth_set(const char *val, const struct kernel_param
>> *kp);
>>   static const struct kernel_param_ops io_queue_depth_ops = {
>>         .set = io_queue_depth_set,
>> @@ -178,6 +184,7 @@ struct nvme_queue {
>>   struct nvme_iod {
>>         struct nvme_request req;
>>         struct nvme_queue *nvmeq;
>> +       bool use_sgl;
>>         int aborted;
>>         int npages;             /* In the PRP list. 0 means small pool in
>> use */
>>         int nents;              /* Used in scatterlist */
>> @@ -331,17 +338,32 @@ static int nvme_npages(unsigned size, struct
>> nvme_dev *dev)
>>         return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
>>   }
>>   +/*
>> + * Calculates the number of pages needed for the SGL segments. For
>> example a 4k
>> + * page can accommodate 256 SGL descriptors.
>> + */
>> +static int nvme_npages_sgl(unsigned int num_seg)
>> +{
>> +       return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc),
>> PAGE_SIZE);
>> +}
>> +
>>   static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
>> -               unsigned int size, unsigned int nseg)
>> +               unsigned int size, unsigned int nseg, bool use_sgl)
>>   {
>> -       return sizeof(__le64 *) * nvme_npages(size, dev) +
>> -                       sizeof(struct scatterlist) * nseg;
>> +       size_t alloc_size;
>> +
>> +       if (use_sgl)
>> +               alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
>> +       else
>> +               alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
>> +
>> +       return alloc_size + sizeof(struct scatterlist) * nseg;
>>   }
>>   -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
>> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>>   {
>>         return sizeof(struct nvme_iod) +
>> -               nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev),
>> NVME_INT_PAGES);
>> +               nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev),
>> NVME_INT_PAGES, use_sgl);
>>   }
>>     static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void
>> *data,
>> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue
>> *nvmeq,
>>         nvmeq->sq_tail = tail;
>>   }
>>   -static __le64 **iod_list(struct request *req)
>> +static void **iod_list(struct request *req)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -       return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
>> +       return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>>   }
>>     static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev
>> *dev)
>> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq,
>> struct nvme_dev *dev)
>>         unsigned int size = blk_rq_payload_bytes(rq);
>>         if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
>> -               iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg),
>> GFP_ATOMIC);
>> +               size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
>> +                               iod->use_sgl);2. Remove the prp suffix in the dev->prp_small_pool and dev->prp_page_pool in all places
    of code.
>> +
>> +               iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>>                 if (!iod->sg)
>>                         return BLK_STS_RESOURCE;
>>         } else {
>> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request
>> *rq, struct nvme_dev *dev)
>>   static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -       const int last_prp = dev->ctrl.page_size / 8 - 1;
>> +       const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
>> +       dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>>         int i;
>> -       __le64 **list = iod_list(req);
>> -       dma_addr_t prp_dma = iod->first_dma;
>>         if (iod->npages == 0)
>> -               dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
>> +               dma_pool_free(dev->prp_small_pool, iod_list(req)[0],
>> dma_addr);
>> +
>>         for (i = 0; i < iod->npages; i++) {
>> -               __le64 *prp_list = list[i];
>> -               dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
>> -               dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
>> -               prp_dma = next_prp_dma;
>> +               void *addr = iod_list(req)[i];
>> +
>> +               if (iod->use_sgl) {
>> +                       struct nvme_sgl_desc *sg_list = addr;
>> +
>> +                       next_dma_addr =
>> +                               le64_to_cpu((sg_list[SGES_PER_PAGE -
>> 1]).addr);
>> +               } else {
>> +                       __le64 *prp_list = addr;
>> +
>> +                       next_dma_addr = le64_to_cpu(prp_list[last_prp]);
>> +               }
>> +
>> +               dma_pool_free(dev->prp_page_pool, addr, dma_addr);
>> +               dma_addr = next_dma_addr;
>>         }
>>         if (iod->sg != iod->inline_sg)
>> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl,
>> int nents)
>>         }
>>   }
>>   -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct
>> request *req)
>> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request
>> *req,
>> +               struct nvme_rw_command *cmnd)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>>         struct dma_pool *pool;
>> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         u32 page_size = dev->ctrl.page_size;
>>         int offset = dma_addr & (page_size - 1);
>>         __le64 *prp_list;
>> -       __le64 **list = iod_list(req);
>> +       void **list = iod_list(req);
>>         dma_addr_t prp_dma;
>>         int nprps, i;
>>   +     iod->use_sgl = false;
>> +
>>         length -= (page_size - offset);
>>         if (length <= 0) {
>>                 iod->first_dma = 0;
>> -               return BLK_STS_OK;
>> +               goto done;
>>         }
>>         dma_len -= (page_size - offset);
>> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         if (length <= page_size) {
>>                 iod->first_dma = dma_addr;
>> -               return BLK_STS_OK;
>> +               goto done;
>>         }
>>         nprps = DIV_ROUND_UP(length, page_size);
>> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>                 dma_len = sg_dma_len(sg);
>>         }
>>   +done:
>> +       cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
>> +       cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
>> +
>>         return BLK_STS_OK;
>>      bad_sgl:
>> @@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         return BLK_STS_IOERR;
>>   }
>>   +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
>> +               struct scatterlist *sg)
>> +{
>> +       sge->addr = cpu_to_le64(sg_dma_address(sg));
>> +       sge->length = cpu_to_le32(sg_dma_len(sg));
>> +       sge->type = NVME_SGL_FMT_DATA_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
>> +               dma_addr_t dma_addr, int entries)
>> +{
>> +       sge->addr = cpu_to_le64(dma_addr);
>> +       sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
>> +       sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
>> +               dma_addr_t dma_addr)
>> +{
>> +       sge->addr = cpu_to_le64(dma_addr);
>> +       sge->length = cpu_to_le32(PAGE_SIZE);
>> +       sge->type = NVME_SGL_FMT_SEG_DESC << 4;
>> +}
>> +
>> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request
>> *req,
>> +               struct nvme_rw_command *cmd)
>> +{
>> +       struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> +       int length = blk_rq_payload_bytes(req);
>> +       struct dma_pool *pool;
>> +       struct nvme_sgl_desc *sg_list;
>> +       struct scatterlist *sg = iod->sg;
>> +       int entries = iod->nents, i = 0;
>> +       dma_addr_t sgl_dma;
>> +
>> +       iod->use_sgl = true;
>> +
>> +       cmd->flags = NVME_CMD_SGL_METABUF; /* setting the transfer type as
>> SGL */
>> +
>> +       if (length == sg_dma_len(sg)) {
>> +               nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
>> +               return BLK_STS_OK;
>> +       }
>> +
>> +       if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
>
>
> where does 256 comes from ? can we use macro definition here ?
[CK] The function nvme_setup_prp_pools() uses 256 literal value
to setup the small dma pool. This value is also used as a literal
in the function nvme_setup_prps(). In order to keep the implementation
of the nvme_setup_sgls() similar to the nvme_setup_prps() when
calculating the number of entries with as a literal we use 256.

Also if you observed we are also using dev->prp_small_pool and
dev->prp_page_pool in nvme_setup_sgls() in this patch.

The purpose of this patch to add the SGL support without introducing any
cleanup in the existing code to make it easier to review.

I'll be happy to send a separate cleanup patch with following changes once
this patch gets accepted:-

1. Remove the 256 value and use the
    NVME_PCI_DMA_POOL_SMALL_BLK_SIZE or any other macro name that
    everyone prefers.
2. Remove the prp suffix in the dev->prp_small_pool and dev->prp_page_pool
    in all places of code.
>
>
>> +               pool = dev->prp_small_pool;
>> +               iod->npages = 0;
>> +       } else {
>> +               pool = dev->prp_page_pool;
>> +               iod->npages = 1;
>> +       }
>> +
>> +       sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
>> +       if (!sg_list) {
>> +               iod->npages = -1;
>> +               return BLK_STS_RESOURCE;
>> +       }
>> +
>> +       iod_list(req)[0] = sg_list;
>> +       iod->first_dma = sgl_dma;
>> +
>> +       if (entries <= SGES_PER_PAGE) {
>> +               nvme_pci_sgl_set_last_seg(&cmd->dptr.sgl, sgl_dma,
>> entries);
>> +
>> +               for (i = 0; i < entries; i++) {
>> +                       nvme_pci_sgl_set_data(&sg_list[i], sg);
>> +                       length -= sg_dma_len(sg);
>> +                       sg = sg_next(sg);
>> +               }
>> +
>> +               WARN_ON(length > 0);
>> +               return BLK_STS_OK;
>> +       }
>> +
>> +       nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma);
>> +
>> +       do {
>> +               if (i == SGES_PER_PAGE) {
>> +                       struct nvme_sgl_desc *old_sg_desc = sg_list;
>> +                       struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
>> +
>> +                       sg_list = dma_pool_alloc(pool, GFP_ATOMIC,
>> &sgl_dma);
>> +                       if (!sg_list)
>> +                               return BLK_STS_RESOURCE;
>> +
>> +                       i = 0;
>> +                       iod_list(req)[iod->npages++] = sg_list;
>> +                       sg_list[i++] = *link;
>> +
>> +                       if (entries < SGES_PER_PAGE)
>> +                               nvme_pci_sgl_set_last_seg(link, sgl_dma,
>> entries);
>> +                       else
>> +                               nvme_pci_sgl_set_seg(link, sgl_dma);
>> +               }
>> +
>> +               nvme_pci_sgl_set_data(&sg_list[i], sg);
>> +
>> +               length -= sg_dma_len(sg);
>> +               sg = sg_next(sg);
>> +               entries--;
>> +       } while (length > 0);
>> +
>> +       WARN_ON(entries > 0);
>> +       return BLK_STS_OK;
>> +}
>> +
>> +static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request
>> *req)
>> +{
>> +       struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> +       unsigned int avg_seg_size;
>> +
>> +       avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
>> +                       blk_rq_nr_phys_segments(req));
>> +
>> +       if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
>
>
> Can you create macros for the spec definitions ?
> In case of Dword alignment and granularity, where do we check that
> requirement ?
>
>
>> +               return false;
>> +       if (!iod->nvmeq->qid)
>> +               return false;
>> +       if (!sgl_threshold || avg_seg_size < sgl_threshold)
>> +               return false;
>> +       return true;
>> +}
>> +
>>   static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request
>> *req,
>>                 struct nvme_command *cmnd)
>>   {
>> @@ -662,7 +828,11 @@ static blk_status_t nvme_map_data(struct nvme_dev
>> *dev, struct request *req,
>>                                 DMA_ATTR_NO_WARN))
>>                 goto out;
>>   -     ret = nvme_setup_prps(dev, req);
>> +       if (nvme_pci_use_sgls(dev, req))
>> +               ret = nvme_setup_sgls(dev, req, &cmnd->rw);
>> +       else
>> +               ret = nvme_setup_prps(dev, req, &cmnd->rw);
>> +
>>         if (ret != BLK_STS_OK)
>>                 goto out_unmap;
>>   @@ -682,8 +852,6 @@ static blk_status_t nvme_map_data(struct nvme_dev
>> *dev, struct request *req,
>>                         goto out_unmap;
>>         }
>>   -     cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
>> -       cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
>>         if (blk_integrity_rq(req))
>>                 cmnd->rw.metadata =
>> cpu_to_le64(sg_dma_address(&iod->meta_sg));
>>         return BLK_STS_OK;
>> @@ -1379,7 +1547,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev
>> *dev)
>>                 dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
>>                 dev->admin_tagset.timeout = ADMIN_TIMEOUT;
>>                 dev->admin_tagset.numa_node = dev_to_node(dev->dev);
>> -               dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
>> +               dev->admin_tagset.cmd_size = nvme_cmd_size(dev, false);
>>                 dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
>>                 dev->admin_tagset.driver_data = dev;
>>   @@ -1906,7 +2074,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
>>                 dev->tagset.numa_node = dev_to_node(dev->dev);
>>                 dev->tagset.queue_depth =
>>                                 min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH)
>> - 1;
>> -               dev->tagset.cmd_size = nvme_cmd_size(dev);
>> +               dev->tagset.cmd_size = nvme_cmd_size(dev, false);
>> +               if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) &&
>> sgl_threshold) {
>> +                       dev->tagset.cmd_size = max(dev->tagset.cmd_size,
>> +                                       nvme_cmd_size(dev, true));
>> +               }
>>                 dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
>>                 dev->tagset.driver_data = dev;
>>


From ckulkarnilinux at gmail.com  Thu Oct 12 20:35:44 2017
From: ckulkarnilinux at gmail.com (chaitany kulkarni)
Date: Thu, 12 Oct 2017 20:35:44 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <d06ea87b-1b0c-e581-b6f1-b4b2647eb431@grimberg.me>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
 <d06ea87b-1b0c-e581-b6f1-b4b2647eb431@grimberg.me>
Message-ID: <CAK0tqi=cx8NGiD_3pfk0R1JtrytO99AUoNy6d0HSd8wJhvyArA@mail.gmail.com>

On Wed, Oct 11, 2017 at 4:01 AM, Sagi Grimberg <sagi at grimberg.me> wrote:
> Chaitanya,
>
> Sorry for the late reply, was on vacation...
>
>> From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>>
>> This adds SGL support for NVMe PCIe driver, based on an earlier patch
>> from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
>> refactors the original code and adds new module parameter sgl_threshold
>> to determine whether to use SGL or PRP for IOs.
>
>
> One of the more appealing values of SGL support is removing the
> virt_boundary constraint from the driver and use SGLs to issue
> "gappy" IO. I would be very happy to see this happening, is this
> something you plan to address?

[CK]Yes, once this patch gets accepted I'm planning to add the
support for gappy IOs and remove the virt_boundary constraints.

>
> This is why I wanted to not have a threshold based sgl/prp selection
> as it does not allow us to remove the virt_boundary constraint for
> controllers that don't have the "prp-able limitation".
>
> We could have also a threshold to select the optimal selection for
> sgl/prp, but it would need to meet the following conditions:
> - choose prp if
>    1. avg_seg_size is < sgl_threshold
>    2. bio_vec is not gappy
[CK] Yes, we should do this ultimately instead of getting rid of the
         threshold parameter.

> (2) would mean traversing the bio_vec until we integrate this into
>     the block layer. But given that this is cheap when the bio_vec
>     is small and long bio_vecs its probably negligible compared to
>     the longer transfer latency.
[CK] Agree.
>>
>> +/*
>> + * Calculates the number of pages needed for the SGL segments. For
>> example a 4k
>> + * page can accommodate 256 SGL descriptors.
>> + */
>> +static int nvme_npages_sgl(unsigned int num_seg)
>
>
> Can you place nvme_pci_ prefix on all the new routines you add?
[CK] Sure.
>
>> +{
>> +       return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc),
>> PAGE_SIZE);
>> +}
>> +
>>   static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
>> -               unsigned int size, unsigned int nseg)
>> +               unsigned int size, unsigned int nseg, bool use_sgl)
>>   {
>
>
> I would suggest that also for functions that you touch their signature
> make them nvme_pci_*.
[CK] Sure, I'll make this change in the next version of this patch.
>
>
>> -       return sizeof(__le64 *) * nvme_npages(size, dev) +
>> -                       sizeof(struct scatterlist) * nseg;
>> +       size_t alloc_size;
>> +
>> +       if (use_sgl)
>> +               alloc_size = sizeof(__le64 *) * nvme_npages_sgl(nseg);
>> +       else
>> +               alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
>> +
>> +       return alloc_size + sizeof(struct scatterlist) * nseg;
>>   }
>>   -static unsigned int nvme_cmd_size(struct nvme_dev *dev)
>> +static unsigned int nvme_cmd_size(struct nvme_dev *dev, bool use_sgl)
>>   {
>>         return sizeof(struct nvme_iod) +
>> -               nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev),
>> NVME_INT_PAGES);
>> +               nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev),
>> NVME_INT_PAGES, use_sgl);
>>   }
>>     static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void
>> *data,[CK] Noted.
>> @@ -425,10 +447,10 @@ static void __nvme_submit_cmd(struct nvme_queue
>> *nvmeq,
>>         nvmeq->sq_tail = tail;
>>   }
>>   -static __le64 **iod_list(struct request *req)
>> +static void **iod_list(struct request *req)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -       return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
>> +       return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
>>   }
>>     static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev
>> *dev)
>> @@ -438,7 +460,10 @@ static blk_status_t nvme_init_iod(struct request *rq,
>> struct nvme_dev *dev)
>>         unsigned int size = blk_rq_payload_bytes(rq);
>>         if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
>> -               iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg),
>> GFP_ATOMIC);
>> +               size_t alloc_size = nvme_iod_alloc_size(dev, size, nseg,
>> +                               iod->use_sgl);
>> +
>> +               iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
>>                 if (!iod->sg)
>>                         return BLK_STS_RESOURCE;
>>         } else {
>> @@ -456,18 +481,29 @@ static blk_status_t nvme_init_iod(struct request
>> *rq, struct nvme_dev *dev)
>>   static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>> -       const int last_prp = dev->ctrl.page_size / 8 - 1;
>> +       const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
>> +       dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
>>         int i;
>> -       __le64 **list = iod_list(req);
>> -       dma_addr_t prp_dma = iod->first_dma;
>>         if (iod->npages == 0)
>> -               dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
>> +               dma_pool_free(dev->prp_small_pool, iod_list(req)[0],
>> dma_addr);
>> +
>>         for (i = 0; i < iod->npages; i++) {
>> -               __le64 *prp_list = list[i];
>> -               dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
>> -               dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
>> -               prp_dma = next_prp_dma;
>> +               void *addr = iod_list(req)[i];
>> +
>> +               if (iod->use_sgl) {
>> +                       struct nvme_sgl_desc *sg_list = addr;
>> +
>> +                       next_dma_addr =
>> +                               le64_to_cpu((sg_list[SGES_PER_PAGE -
>> 1]).addr);
>> +               } else {
>> +                       __le64 *prp_list = addr;
>> +
>> +                       next_dma_addr = le64_to_cpu(prp_list[last_prp]);
>> +               }
>> +
>> +               dma_pool_free(dev->prp_page_pool, addr, dma_addr);
>> +               dma_addr = next_dma_addr;
>>         }
>>         if (iod->sg != iod->inline_sg)
>> @@ -555,7 +591,8 @@ static void nvme_print_sgl(struct scatterlist *sgl,
>> int nents)
>>         }
>>   }
>>   -static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct
>> request *req)
>> +static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request
>> *req,
>> +               struct nvme_rw_command *cmnd)
>>   {
>>         struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>>         struct dma_pool *pool;
>> @@ -566,14 +603,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         u32 page_size = dev->ctrl.page_size;
>>         int offset = dma_addr & (page_size - 1);
>>         __le64 *prp_list;
>> -       __le64 **list = iod_list(req);
>> +       void **list = iod_list(req);
>>         dma_addr_t prp_dma;
>>         int nprps, i;
>>   +     iod->use_sgl = false;
>> +
>>         length -= (page_size - offset);
>>         if (length <= 0) {
>>                 iod->first_dma = 0;
>> -               return BLK_STS_OK;
>> +               goto done;
>>         }
>>         dma_len -= (page_size - offset);
>> @@ -587,7 +626,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         if (length <= page_size) {
>>                 iod->first_dma = dma_addr;
>> -               return BLK_STS_OK;
>> +               goto done;
>>         }
>>         nprps = DIV_ROUND_UP(length, page_size);
>> @@ -634,6 +673,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>                 dma_len = sg_dma_len(sg);
>>         }
>>   +done:
>> +       cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
>> +       cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
>> +
>>         return BLK_STS_OK;
>>      bad_sgl:
>> @@ -643,6 +686,129 @@ static blk_status_t nvme_setup_prps(struct nvme_dev
>> *dev, struct request *req)
>>         return BLK_STS_IOERR;
>>   }
>>   +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
>> +               struct scatterlist *sg)
>> +{
>> +       sge->addr = cpu_to_le64(sg_dma_address(sg));
>> +       sge->length = cpu_to_le32(sg_dma_len(sg));
>> +       sge->type = NVME_SGL_FMT_DATA_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_last_seg(struct nvme_sgl_desc *sge,
>> +               dma_addr_t dma_addr, int entries)
>> +{
>> +       sge->addr = cpu_to_le64(dma_addr);
>> +       sge->length = cpu_to_le32(entries * sizeof(struct nvme_sgl_desc));
>> +       sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
>> +}
>> +
>> +static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
>> +               dma_addr_t dma_addr)
>> +{
>> +       sge->addr = cpu_to_le64(dma_addr);
>> +       sge->length = cpu_to_le32(PAGE_SIZE);
>> +       sge->type = NVME_SGL_FMT_SEG_DESC << 4;
>> +}
>> +
>> +static blk_status_t nvme_setup_sgls(struct nvme_dev *dev, struct request
>> *req,
>> +               struct nvme_rw_command *cmd)
>> +{
>
>
> nvme_pci_*
[CK] Noted.


From jthumshirn at suse.de  Fri Oct 13 01:41:48 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:41:48 +0200
Subject: [PATCH 3/8] nvmet/fc: Use sgl_alloc() and sgl_free()
In-Reply-To: <20171012224550.25440-4-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-4-bart.vanassche@wdc.com>
Message-ID: <20171013084148.ua4akc2fwxooxrye@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 13 01:42:14 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:42:14 +0200
Subject: [PATCH 4/8] nvmet/rdma: Use sgl_alloc() and sgl_free()
In-Reply-To: <20171012224550.25440-5-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-5-bart.vanassche@wdc.com>
Message-ID: <20171013084214.zdfocnxzfyncnom3@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 13 01:44:39 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:44:39 +0200
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <1507849209.14112.1.camel@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
 <ee12eeaf-9848-478d-4ebe-9d3497e248e0@kernel.dk>
 <1507849209.14112.1.camel@wdc.com>
Message-ID: <20171013084439.mgidqaygf2kbuz5u@linux-x5ow.site>

On Thu, Oct 12, 2017 at 11:00:10PM +0000, Bart Van Assche wrote:
> On Thu, 2017-10-12 at 16:52 -0600, Jens Axboe wrote:
> > On 10/12/2017 04:45 PM, Bart Van Assche wrote:
> > > +++ b/include/linux/sgl_alloc.h
> > > @@ -0,0 +1,16 @@
> > > +#ifndef _SGL_ALLOC_H_
> > > +#define _SGL_ALLOC_H_
> > > +
> > > +#include <linux/types.h> /* bool, gfp_t */
> > > +
> > > +struct scatterlist;
> > > +
> > > +struct scatterlist *sgl_alloc_order(unsigned long long length,
> > > +				    unsigned int order, unsigned int *nent_p,
> > > +				    gfp_t gfp, bool chainable);
> > > +struct scatterlist *sgl_alloc(unsigned long long length, unsigned int *nent_p,
> > > +			      gfp_t gfp);
> > > +void sgl_free_order(struct scatterlist *sgl, int order);
> > > +void sgl_free(struct scatterlist *sgl);
> > > +
> > > +#endif /* _SGL_ALLOC_H_ */
> > 
> > Should this just go into linux/scatterlist.h instead of creating a new header
> > file?
> 
> That's something I can change. I don't have a strong opinion about whether
> these declarations should go into a new header file or into linux/scatterlist.h.

Yes please do (and the implementaion should go into lib/scatterlist.c).
Additionally please get rid of the new CONFIG_SGL_ALLOC as well.

Thanks,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 13 01:45:33 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:45:33 +0200
Subject: [PATCH 6/8] scsi/ipr: Use sgl_alloc_order() and sgl_free_order()
In-Reply-To: <20171012224550.25440-7-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-7-bart.vanassche@wdc.com>
Message-ID: <20171013084532.jd64uwjxl2kb6d2p@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 13 01:45:57 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:45:57 +0200
Subject: [PATCH 7/8] scsi/pmcraid: Remove an unused structure member
In-Reply-To: <20171012224550.25440-8-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-8-bart.vanassche@wdc.com>
Message-ID: <20171013084556.nm74pesdsn6aejfb@linux-x5ow.site>

Fair enough,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 13 01:47:18 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 13 Oct 2017 10:47:18 +0200
Subject: [PATCH 8/8] scsi/pmcraid: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <20171012224550.25440-9-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-9-bart.vanassche@wdc.com>
Message-ID: <20171013084717.7odpun36aywlvbvd@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From axboe at kernel.dk  Fri Oct 13 07:20:57 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Fri, 13 Oct 2017 08:20:57 -0600
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <1507849209.14112.1.camel@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
 <ee12eeaf-9848-478d-4ebe-9d3497e248e0@kernel.dk>
 <1507849209.14112.1.camel@wdc.com>
Message-ID: <f8e331f3-d77c-0da2-208e-5d3f68d50565@kernel.dk>

On 10/12/2017 05:00 PM, Bart Van Assche wrote:
> On Thu, 2017-10-12 at 16:52 -0600, Jens Axboe wrote:
>> On 10/12/2017 04:45 PM, Bart Van Assche wrote:
>>> +++ b/include/linux/sgl_alloc.h
>>> @@ -0,0 +1,16 @@
>>> +#ifndef _SGL_ALLOC_H_
>>> +#define _SGL_ALLOC_H_
>>> +
>>> +#include <linux/types.h> /* bool, gfp_t */
>>> +
>>> +struct scatterlist;
>>> +
>>> +struct scatterlist *sgl_alloc_order(unsigned long long length,
>>> +				    unsigned int order, unsigned int *nent_p,
>>> +				    gfp_t gfp, bool chainable);
>>> +struct scatterlist *sgl_alloc(unsigned long long length, unsigned int *nent_p,
>>> +			      gfp_t gfp);
>>> +void sgl_free_order(struct scatterlist *sgl, int order);
>>> +void sgl_free(struct scatterlist *sgl);
>>> +
>>> +#endif /* _SGL_ALLOC_H_ */
>>
>> Should this just go into linux/scatterlist.h instead of creating a new header
>> file?
> 
> That's something I can change. I don't have a strong opinion about whether
> these declarations should go into a new header file or into linux/scatterlist.h.

I think you should make that change, keep things simpler. As Johannes
suggested, you could move the code to lib/scatterlist.c as well. I would
recommend keeping the kconfig option to only build it when needed, though.

-- 
Jens Axboe



From rdunlap at infradead.org  Fri Oct 13 10:43:43 2017
From: rdunlap at infradead.org (Randy Dunlap)
Date: Fri, 13 Oct 2017 10:43:43 -0700
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <20171012224550.25440-2-bart.vanassche@wdc.com>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
Message-ID: <a249d319-83fa-d2c3-5723-21de238787d5@infradead.org>

On 10/12/17 15:45, Bart Van Assche wrote:

> diff --git a/lib/sgl_alloc.c b/lib/sgl_alloc.c
> new file mode 100644
> index 000000000000..d96b395dd5c8
> --- /dev/null
> +++ b/lib/sgl_alloc.c
> @@ -0,0 +1,102 @@

> +/**
> + * sgl_free_order - free a scatterlist and its pages
> + * @sg: Scatterlist with one or more elements

      @sgl:

> + * @order: Second argument for __free_pages()
> + */
> +void sgl_free_order(struct scatterlist *sgl, int order)
> +{
> +	struct scatterlist *sg;
> +	struct page *page;
> +
> +	for (sg = sgl; sg; sg = sg_next(sg)) {
> +		page = sg_page(sg);
> +		if (page)
> +			__free_pages(page, order);
> +	}
> +	kfree(sgl);
> +}
> +EXPORT_SYMBOL(sgl_free_order);
> +
> +/**
> + * sgl_free - free a scatterlist and its pages
> + * @sg: Scatterlist with one or more elements

      @sgl:

> + */
> +void sgl_free(struct scatterlist *sgl)
> +{
> +	sgl_free_order(sgl, 0);
> +}
> +EXPORT_SYMBOL(sgl_free);
> 

ta.
-- 
~Randy


From Bart.VanAssche at wdc.com  Fri Oct 13 10:56:38 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Fri, 13 Oct 2017 17:56:38 +0000
Subject: [PATCH 1/8] lib: Introduce sgl_alloc() and sgl_free()
In-Reply-To: <a249d319-83fa-d2c3-5723-21de238787d5@infradead.org>
References: <20171012224550.25440-1-bart.vanassche@wdc.com>
 <20171012224550.25440-2-bart.vanassche@wdc.com>
 <a249d319-83fa-d2c3-5723-21de238787d5@infradead.org>
Message-ID: <1507917395.13345.28.camel@wdc.com>

On Fri, 2017-10-13 at 10:43 -0700, Randy Dunlap wrote:
> On 10/12/17 15:45, Bart Van Assche wrote:
> > + * @sg: Scatterlist with one or more elements
> 
>       @sgl:

Thanks Randy. I will fix this before I repost this series.

Bart.

From jsmart2021 at gmail.com  Fri Oct 13 13:38:00 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 13 Oct 2017 13:38:00 -0700
Subject: [PATCH] nvmet: protect sqhd update by a lock
Message-ID: <20171013203800.24125-1-jsmart2021@gmail.com>

In testing target io in read write mix, we did indeed get into cases
where sqhd didn't update properly and slowly missed enough updates to
shutdown the queue.

Protect the read/modify/set update of sqhd under a lock for coherency.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/target/core.c  | 8 +++++++-
 drivers/nvme/target/nvmet.h | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index cfc35cbb6fe2..c753826390d6 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -387,11 +387,16 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 static void __nvmet_req_complete(struct nvmet_req *req, u16 status)
 {
+	unsigned long flags;
+
 	if (status)
 		nvmet_set_status(req, status);
 
-	if (req->sq->size)
+	if (req->sq->size) {
+		spin_lock_irqsave(&req->sq->sqhd_lock, flags);
 		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
+		spin_unlock_irqrestore(&req->sq->sqhd_lock, flags);
+	}
 	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
 	req->rsp->sq_id = cpu_to_le16(req->sq->qid);
 	req->rsp->command_id = req->cmd->common.command_id;
@@ -425,6 +430,7 @@ void nvmet_sq_setup(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq,
 	sq->size = size;
 
 	ctrl->sqs[qid] = sq;
+	spin_lock_init(&sq->sqhd_lock);
 }
 
 static void nvmet_confirm_sq(struct percpu_ref *ref)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index ed38b44a7007..bdb904bdf676 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -77,6 +77,7 @@ struct nvmet_sq {
 	u16			sqhd;
 	struct completion	free_done;
 	struct completion	confirm_done;
+	spinlock_t		sqhd_lock;
 };
 
 /**
-- 
2.13.1



From maxg at mellanox.com  Sat Oct 14 11:41:45 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sat, 14 Oct 2017 21:41:45 +0300
Subject: [PATCH] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
References: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
Message-ID: <0ee63b18-9b42-a2c9-be1a-74c69735eb87@mellanox.com>



On 10/10/2017 6:59 PM, roys at lightbitslabs.com wrote:
> From: Roy Shterman <roys at lightbitslabs.com>
> 
> In case we disable namespaces which has the nsid like
> subsystem max_nsid we need to search for the next largest nsid
> in this subsystem. If the subsystem don't has more namespaces
> we set it to 0, else we take nsid from the last namespace in
> namespaces list because the list is sorted while inserting.
> 
> Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
> ---

Looks good,

Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From nitzanc at mellanox.com  Sun Oct 15 01:05:50 2017
From: nitzanc at mellanox.com (Nitzan Carmi)
Date: Sun, 15 Oct 2017 11:05:50 +0300
Subject: [PATCH 6/7] nvme-rdma: teardown admin/io queues once on error
 recovery
In-Reply-To: <1507724952-13917-7-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
 <1507724952-13917-7-git-send-email-sagi@grimberg.me>
Message-ID: <cee6d667-af6c-71c4-e996-96f907e8ef29@mellanox.com>



On 11/10/2017 15:29, Sagi Grimberg wrote:
> Relying on the queue state while tearing down on every reconnect
> attempt is not a good design. We should do it once in err_work
> and simply try to establish the queues for each reconnect attempt.
>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   drivers/nvme/host/rdma.c | 27 ++++++++++++---------------
>   1 file changed, 12 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index e3fe57011fcb..2a18465bf361 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -925,10 +925,6 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
>   
>   	++ctrl->ctrl.nr_reconnects;
>   
> -	if (ctrl->ctrl.queue_count > 1)
> -		nvme_rdma_destroy_io_queues(ctrl, false);
> -
> -	nvme_rdma_destroy_admin_queue(ctrl, false);
>   	ret = nvme_rdma_configure_admin_queue(ctrl, false);
>   	if (ret)
>   		goto requeue;
> @@ -936,7 +932,7 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
>   	if (ctrl->ctrl.queue_count > 1) {
>   		ret = nvme_rdma_configure_io_queues(ctrl, false);
>   		if (ret)
> -			goto requeue;
> +			goto destroy_admin;
>   	}
>   
>   	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
> @@ -946,14 +942,17 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
>   		return;
>   	}
>   
> -	ctrl->ctrl.nr_reconnects = 0;
> -
>   	nvme_start_ctrl(&ctrl->ctrl);
>   
> -	dev_info(ctrl->ctrl.device, "Successfully reconnected\n");
> +	dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attepmpt)\n",
typo :)
> +			ctrl->ctrl.nr_reconnects);
> +
> +	ctrl->ctrl.nr_reconnects = 0;
>   
>   	return;
>   
> +destroy_admin:
> +	nvme_rdma_destroy_admin_queue(ctrl, false);
>   requeue:
>   	dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
>   			ctrl->ctrl.nr_reconnects);
> @@ -969,17 +968,15 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
>   
>   	if (ctrl->ctrl.queue_count > 1) {
>   		nvme_stop_queues(&ctrl->ctrl);
> -		nvme_rdma_stop_io_queues(ctrl);
> -	}
> -	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
> -	nvme_rdma_stop_queue(&ctrl->queues[0]);
> -
> -	/* We must take care of fastfail/requeue all our inflight requests */
> -	if (ctrl->ctrl.queue_count > 1)
>   		blk_mq_tagset_busy_iter(&ctrl->tag_set,
>   					nvme_cancel_request, &ctrl->ctrl);
> +		nvme_rdma_destroy_io_queues(ctrl, false);
> +	}
> +
> +	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
>   	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
>   				nvme_cancel_request, &ctrl->ctrl);
> +	nvme_rdma_destroy_admin_queue(ctrl, false);
>   
>   	/*
>   	 * queues are not a live anymore, so restart the queues to fail fast



From maxg at mellanox.com  Sun Oct 15 06:25:59 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sun, 15 Oct 2017 16:25:59 +0300
Subject: [PATCH 3/4] nvmet: Add controllers to configfs
In-Reply-To: <834d07fa-5830-ea7a-11b1-f20ebf47831c@gmail.com>
References: <1507635633-16665-1-git-send-email-israelr@mellanox.com>
 <1507635633-16665-4-git-send-email-israelr@mellanox.com>
 <834d07fa-5830-ea7a-11b1-f20ebf47831c@gmail.com>
Message-ID: <923a51f0-61f7-7813-97fc-65bf072a9bb6@mellanox.com>



On 10/15/2017 11:49 AM, roy wrote:
> Hi,
> 
> I think there is a potential problem here. for example in the next scenario:
> 
> core X is reading .../controllers/N/hostnqn and core Y is handling 
> nvmet_ctrl_free. basically
> 
> core Y can free the ctrl while core X will try to do sprintf on 
> ctrl->hostnqn which will cause NULL dereference.
> 
> Am I missing something?
> 

we delete the ctrl_configfs before freeing the ctrl. parallel processing 
should be protected by internal configfs mutex, just like in the case of 
reading namespace or port items.

for example:

static ssize_t nvmet_ns_enable_show(struct config_item *item, char *page)
{
         return sprintf(page, "%d\n", to_nvmet_ns(item)->enabled);
}

What I do see is an option for freeing the port (by rmdir ../ports/1/) 
and then ctrl->port will point to freed memory. I guess we can cache the 
needed values we want to show.


From jthumshirn at suse.de  Mon Oct 16 00:21:45 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 16 Oct 2017 09:21:45 +0200
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <20171013203800.24125-1-jsmart2021@gmail.com>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
Message-ID: <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>

To me it looks a bit awkward why you're only protecting the RMW part but not
the re-read below.

Could you please give some explanation why this is not needed?

Thanks,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at infradead.org  Mon Oct 16 00:58:45 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 00:58:45 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
 <a04ca9b1-c551-d86e-2e61-fc37c7ddd086@mellanox.com>
Message-ID: <20171016075845.GA14943@infradead.org>

[can you trim down your quotes to the relevant part?  That would
 make it a lot easier to read]

On Sun, Oct 08, 2017 at 12:16:10PM +0300, Max Gurtovoy wrote:
> >   #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
> > +#define SGES_PER_PAGE          (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
> 
> should we use PAGE_SIZE or ctrl->page_size ?

It should be PAGE_SIZE - as that is the size of the PRP page pool
that is also reused for SGLs.

> > +	if (entries <= 256 / sizeof(struct nvme_sgl_desc)) {
>
> 
> where does 256 comes from ? can we use macro definition here ?

That is the size of the small PRP pool.

I think a little cleanup to define constants for the small and
large pools, and remove the PRP from the name might be nice, but
we can do that after this patch.

> > +	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
> 
> Can you create macros for the spec definitions ?
> In case of Dword alignment and granularity, where do we check that
> requirement ?

These bits don't have names in the spec, so I think we should not
make them up.


From hch at infradead.org  Mon Oct 16 00:59:43 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 00:59:43 -0700
Subject: [PATCH V6] nvme-pci: add SGL support
In-Reply-To: <20171009180300.GD19329@localhost.localdomain>
References: <1507232903-9532-1-git-send-email-ckulkarnilinux@gmail.com>
 <1507232903-9532-2-git-send-email-ckulkarnilinux@gmail.com>
 <20171009180300.GD19329@localhost.localdomain>
Message-ID: <20171016075943.GB14943@infradead.org>

On Mon, Oct 09, 2017 at 12:03:00PM -0600, Keith Busch wrote:
> There's two loops here doing essentially the same thing, but I don't
> think the single segment needs to have a special case.

Except that for the larger case we'd need another branch for the
indirection entry.  I'm not sure it's clear cut either way, so I'd
just let Chaitanya chose which one he prefers.


From sagi at grimberg.me  Mon Oct 16 03:21:36 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 16 Oct 2017 13:21:36 +0300
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
Message-ID: <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>

>> Lets take this one step at a time, starting with this issue.
>>
>> First, there is a reason why a simple create_ctrl fails, can we isolate
>> exactly which call fails? Was something else going on that might have
>> made the simple create_ctrl fail?
>>
>> We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
>> messages but we do see "creating I/O queues" which means that we either
>> failed at IO tagset allocation or initializing connect_q.
>>
>> We have a missing error code assignment so can you try the following 
>> patch:
> 
> Hi Sagi
> Sorry for the late response, here is the log with your patch, let me 
> know if you need more test.

Hi Yi,

I was referring to the bug you reported on a simple create_ctrl failed:
https://pastebin.com/7z0XSGSd

Does it reproduce?


From sagi at grimberg.me  Mon Oct 16 03:34:07 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 16 Oct 2017 13:34:07 +0300
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <20171013203800.24125-1-jsmart2021@gmail.com>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
Message-ID: <982bd969-c099-fc6c-9fb5-11884abd8da8@grimberg.me>


> In testing target io in read write mix, we did indeed get into cases
> where sqhd didn't update properly and slowly missed enough updates to
> shutdown the queue.
> 
> Protect the read/modify/set update of sqhd under a lock for coherency.

How does this fix the issue given that updates are not protected?


From hch at infradead.org  Mon Oct 16 05:46:21 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 05:46:21 -0700
Subject: [PATCH 4.14-rc] nvme-rdma: stop controller reset if the
 controller is deleting
In-Reply-To: <1507715391-430-1-git-send-email-sagi@grimberg.me>
References: <1507715391-430-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171016124621.GA26833@infradead.org>

On Wed, Oct 11, 2017 at 12:49:51PM +0300, Sagi Grimberg wrote:
> If the controller is deleting (in case the user decided to delete
> it), we have no point to continue reset sequence.
> 
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

Looks good and mirrors what we do in reconnect.

I'll apply it to nvme-4.14.


From hch at infradead.org  Mon Oct 16 05:48:17 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 05:48:17 -0700
Subject: [PATCH v2 0/3] Move tagset reinit to its only current consumer,
 nvme-core
In-Reply-To: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171016124817.GB26833@infradead.org>

Jens, are you fine with picking this up throught the nvme tree?
I think Sagi's later changes rely on it, so that would make life
a littler easier.


From hch at infradead.org  Mon Oct 16 05:48:49 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 05:48:49 -0700
Subject: [PATCH] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
References: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
Message-ID: <20171016124849.GC26833@infradead.org>

Looks good and a reasonable 4.14. candidate.


From hch at infradead.org  Mon Oct 16 05:49:29 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 05:49:29 -0700
Subject: [PATCH v3] nvme_fc: fix iowait hang
In-Reply-To: <20171009203944.3541-1-jsmart2021@gmail.com>
References: <20171009203944.3541-1-jsmart2021@gmail.com>
Message-ID: <20171016124929.GD26833@infradead.org>

Thanks, will apply to nvme-4.14.


From hch at infradead.org  Mon Oct 16 05:50:40 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 05:50:40 -0700
Subject: [PATCH v2] nvme_fc: retry initial controller connections 3 times
In-Reply-To: <20171009233922.7216-1-jsmart2021@gmail.com>
References: <20171009233922.7216-1-jsmart2021@gmail.com>
Message-ID: <20171016125040.GE26833@infradead.org>

Thanks,

I will apply this to the nvme-4.14 tree.


From hch at infradead.org  Mon Oct 16 06:00:53 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 06:00:53 -0700
Subject: [PATCH] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
References: <1507651167-4839-1-git-send-email-roys@lightbitslabs.com>
Message-ID: <20171016130053.GA1936@infradead.org>

Actually, this doesn't even compile..  the entry in ->namespaces
is dev_link, not entry.

If you resend please also fix the line > 80 chars, preferably
by adding a new helper that returns the new max_nsid value.


From jthumshirn at suse.de  Mon Oct 16 07:44:28 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 16 Oct 2017 16:44:28 +0200
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <963d2bd1-1193-7961-eafe-862ddf6f7a91@gmail.com>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
 <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>
 <963d2bd1-1193-7961-eafe-862ddf6f7a91@gmail.com>
Message-ID: <20171016144428.ejwo5yr5p7iduftn@linux-x5ow.site>

On Mon, Oct 16, 2017 at 07:37:48AM -0700, James Smart wrote:
> Agree, is a little awkward. But works fine. The re-read will contain at
> least the value that it was updated to under the lock. If it's actually the
> value of a simultaneous update that is one further, that's actually fine too
> - as sqhd isn't 1:1 with a cqe. Yes a cqe must contain the sqhd, but sqhd
> can increment independently from a particuluar cqe. Key is that sqhd itself
> must increment sanely so that whenever stamped in a cqe it's valid.
> 
> I have no problem recutting with the lock around the assignment if desired.

This to me sounds like a lock wasn't the right choice but a memory barrier
would be needed. But OTOH I'm by no means a locking expert. What I _think_ is
going on is, spin_lock_irqsave() is called, all the preempt_disable() and
local_irq_save() (which are expensive) do their work and thus stretch timing
enough that the desired effect is in place.

But this is all guesswork.

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From axboe at kernel.dk  Mon Oct 16 07:47:38 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Mon, 16 Oct 2017 08:47:38 -0600
Subject: [PATCH v2 0/3] Move tagset reinit to its only current consumer,
 nvme-core
In-Reply-To: <20171016124817.GB26833@infradead.org>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
 <20171016124817.GB26833@infradead.org>
Message-ID: <718a34b5-b28e-fc4f-5b42-f961dde53482@kernel.dk>

On 10/16/2017 06:48 AM, Christoph Hellwig wrote:
> Jens, are you fine with picking this up throught the nvme tree?
> I think Sagi's later changes rely on it, so that would make life
> a littler easier.

Yeah that's fine, you can add my reviewed-by to the patches
as well.

-- 
Jens Axboe



From james.smart at broadcom.com  Mon Oct 16 07:57:31 2017
From: james.smart at broadcom.com (James Smart)
Date: Mon, 16 Oct 2017 07:57:31 -0700
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <20171016144428.ejwo5yr5p7iduftn@linux-x5ow.site>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
 <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>
 <963d2bd1-1193-7961-eafe-862ddf6f7a91@gmail.com>
 <20171016144428.ejwo5yr5p7iduftn@linux-x5ow.site>
Message-ID: <db261e81-cd01-c5a3-5e60-a81d6dfc50fe@broadcom.com>

The real issue, where memory barrier isn't enough, is between the value 
of the sqhd increment followed by the modulo. Many of a time I've seen 
interruption in between these points.

As far as assignment of the cqe - as long a sqhd is getting updated 
properly, such that the cqe will return either the value just set? or a 
newer one, even if another cqe returns the same newer value, it's fine 
per the sqhd definition in nvme.

As it's no big deal - I'll repost with the lock moved

-- james


On 10/16/2017 7:44 AM, Johannes Thumshirn wrote:
> On Mon, Oct 16, 2017 at 07:37:48AM -0700, James Smart wrote:
>> Agree, is a little awkward. But works fine. The re-read will contain at
>> least the value that it was updated to under the lock. If it's actually the
>> value of a simultaneous update that is one further, that's actually fine too
>> - as sqhd isn't 1:1 with a cqe. Yes a cqe must contain the sqhd, but sqhd
>> can increment independently from a particuluar cqe. Key is that sqhd itself
>> must increment sanely so that whenever stamped in a cqe it's valid.
>>
>> I have no problem recutting with the lock around the assignment if desired.
> This to me sounds like a lock wasn't the right choice but a memory barrier
> would be needed. But OTOH I'm by no means a locking expert. What I _think_ is
> going on is, spin_lock_irqsave() is called, all the preempt_disable() and
> local_irq_save() (which are expensive) do their work and thus stretch timing
> enough that the desired effect is in place.
>
> But this is all guesswork.
>
> Byte,
> 	Johannes
>



From jthumshirn at suse.de  Mon Oct 16 08:00:39 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 16 Oct 2017 17:00:39 +0200
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <db261e81-cd01-c5a3-5e60-a81d6dfc50fe@broadcom.com>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
 <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>
 <963d2bd1-1193-7961-eafe-862ddf6f7a91@gmail.com>
 <20171016144428.ejwo5yr5p7iduftn@linux-x5ow.site>
 <db261e81-cd01-c5a3-5e60-a81d6dfc50fe@broadcom.com>
Message-ID: <20171016150038.i2sjez2bj2dnxeq2@linux-x5ow.site>

On Mon, Oct 16, 2017 at 07:57:31AM -0700, James Smart wrote:
> The real issue, where memory barrier isn't enough, is between the value of
> the sqhd increment followed by the modulo. Many of a time I've seen
> interruption in between these points.
> 
> As far as assignment of the cqe - as long a sqhd is getting updated
> properly, such that the cqe will return either the value just set? or a
> newer one, even if another cqe returns the same newer value, it's fine per
> the sqhd definition in nvme.
> 
> As it's no big deal - I'll repost with the lock moved

OK, thanks.

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jsmart2021 at gmail.com  Mon Oct 16 07:37:48 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon, 16 Oct 2017 07:37:48 -0700
Subject: [PATCH] nvmet: protect sqhd update by a lock
In-Reply-To: <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>
References: <20171013203800.24125-1-jsmart2021@gmail.com>
 <20171016072145.wyrp2fyw2cx2grpf@linux-x5ow.site>
Message-ID: <963d2bd1-1193-7961-eafe-862ddf6f7a91@gmail.com>

On 10/16/2017 12:21 AM, Johannes Thumshirn wrote:
> To me it looks a bit awkward why you're only protecting the RMW part but not
> the re-read below.
> 
> Could you please give some explanation why this is not needed?
> 
> Thanks,
> 	Johannes
> 

Agree, is a little awkward. But works fine. The re-read will contain at 
least the value that it was updated to under the lock. If it's actually 
the value of a simultaneous update that is one further, that's actually 
fine too - as sqhd isn't 1:1 with a cqe. Yes a cqe must contain the 
sqhd, but sqhd can increment independently from a particuluar cqe. Key 
is that sqhd itself must increment sanely so that whenever stamped in a 
cqe it's valid.

I have no problem recutting with the lock around the assignment if desired.

-- james



From jsmart2021 at gmail.com  Mon Oct 16 08:18:20 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon, 16 Oct 2017 08:18:20 -0700
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
Message-ID: <20171016151820.14766-1-jsmart2021@gmail.com>

In testing target io in read write mix, we did indeed get into cases
where sqhd didn't update properly and slowly missed enough updates to
shutdown the queue.

Protect the read/modify/set update of sqhd under a lock for coherency.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2: move locks so around update and assignment

 drivers/nvme/target/core.c  | 5 +++++
 drivers/nvme/target/nvmet.h | 1 +
 2 files changed, 6 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index cfc35cbb6fe2..28300e9bbef1 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -387,12 +387,16 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 static void __nvmet_req_complete(struct nvmet_req *req, u16 status)
 {
+	unsigned long flags;
+
 	if (status)
 		nvmet_set_status(req, status);
 
+	spin_lock_irqsave(&req->sq->sqhd_lock, flags);
 	if (req->sq->size)
 		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
 	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
+	spin_unlock_irqrestore(&req->sq->sqhd_lock, flags);
 	req->rsp->sq_id = cpu_to_le16(req->sq->qid);
 	req->rsp->command_id = req->cmd->common.command_id;
 
@@ -425,6 +429,7 @@ void nvmet_sq_setup(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq,
 	sq->size = size;
 
 	ctrl->sqs[qid] = sq;
+	spin_lock_init(&sq->sqhd_lock);
 }
 
 static void nvmet_confirm_sq(struct percpu_ref *ref)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index ed38b44a7007..bdb904bdf676 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -77,6 +77,7 @@ struct nvmet_sq {
 	u16			sqhd;
 	struct completion	free_done;
 	struct completion	confirm_done;
+	spinlock_t		sqhd_lock;
 };
 
 /**
-- 
2.13.1



From jthumshirn at suse.de  Mon Oct 16 08:31:26 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 16 Oct 2017 17:31:26 +0200
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
In-Reply-To: <20171016151820.14766-1-jsmart2021@gmail.com>
References: <20171016151820.14766-1-jsmart2021@gmail.com>
Message-ID: <20171016153126.mg6q43nthpp4pjil@linux-x5ow.site>

Still somewhat awkward to read, maybe we can fold in a comment on why the lock
only touches these two lines and isn't used outside of this function anymore.

Anyways,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From Bart.VanAssche at wdc.com  Mon Oct 16 15:23:09 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Mon, 16 Oct 2017 22:23:09 +0000
Subject: v4.14-rc5 NVMeOF regression?
Message-ID: <1508192588.2493.60.camel@wdc.com>

Hello,

It has been a while since I ran any NVMeOF tests. But when I tried to test
the v4.14-rc5 NVMeOF drivers the output shown below appeared. Is this a
known issue? The following test triggered these call stacks:

# srp-test/run_tests -n -f xfs -e deadline -r 60

Thanks,

Bart.

======================================================
======================================================
WARNING: possible circular locking dependency detected
WARNING: possible circular locking dependency detected
4.14.0-rc5-dbg+ #3 Not tainted
4.14.0-rc5-dbg+ #3 Not tainted
------------------------------------------------------
------------------------------------------------------
modprobe/2272 is trying to acquire lock:
modprobe/2272 is trying to acquire lock:
 ("events"){+.+.}, at: [<ffffffff81084185>] flush_workqueue+0x75/0x520
 ("events"){+.+.}, at: [<ffffffff81084185>] flush_workqueue+0x75/0x520

but task is already holding lock:

but task is already holding lock:
 (device_mutex){+.+.}, at: [<ffffffffa05d6bf7>] ib_unregister_client+0x27/0x200 [ib_core]
 (device_mutex){+.+.}, at: [<ffffffffa05d6bf7>] ib_unregister_client+0x27/0x200 [ib_core]

which lock already depends on the new lock.


which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

the existing dependency chain (in reverse order) is:

-> #3 (device_mutex){+.+.}:

-> #3 (device_mutex){+.+.}:
       lock_acquire+0xdc/0x1d0
       lock_acquire+0xdc/0x1d0
       __mutex_lock+0x86/0x990
       __mutex_lock+0x86/0x990
       mutex_lock_nested+0x1b/0x20
       mutex_lock_nested+0x1b/0x20
       ib_register_device+0xa3/0x650 [ib_core]
       ib_register_device+0xa3/0x650 [ib_core]
       mlx4_ib_add+0xcfd/0x1440 [mlx4_ib]
       mlx4_ib_add+0xcfd/0x1440 [mlx4_ib]
       mlx4_add_device+0x45/0xe0 [mlx4_core]
       mlx4_add_device+0x45/0xe0 [mlx4_core]
       mlx4_register_interface+0xa8/0x120 [mlx4_core]
       mlx4_register_interface+0xa8/0x120 [mlx4_core]
       0xffffffffa05b2051
       0xffffffffa05b2051
       do_one_initcall+0x43/0x166
       do_one_initcall+0x43/0x166
       do_init_module+0x5f/0x206
       do_init_module+0x5f/0x206
       load_module+0x26fe/0x2db0
       load_module+0x26fe/0x2db0
       SYSC_finit_module+0xbc/0xf0
       SYSC_finit_module+0xbc/0xf0
       SyS_finit_module+0xe/0x10
       SyS_finit_module+0xe/0x10
       entry_SYSCALL_64_fastpath+0x18/0xad
       entry_SYSCALL_64_fastpath+0x18/0xad

-> #2 (intf_mutex){+.+.}:

-> #2 (intf_mutex){+.+.}:
       lock_acquire+0xdc/0x1d0
       lock_acquire+0xdc/0x1d0
       __mutex_lock+0x86/0x990
       __mutex_lock+0x86/0x990
       mutex_lock_nested+0x1b/0x20
       mutex_lock_nested+0x1b/0x20
       mlx4_register_device+0x30/0xc0 [mlx4_core]
       mlx4_register_device+0x30/0xc0 [mlx4_core]
       mlx4_load_one+0x15f4/0x16f0 [mlx4_core]
       mlx4_load_one+0x15f4/0x16f0 [mlx4_core]
       mlx4_init_one+0x4b9/0x700 [mlx4_core]
       mlx4_init_one+0x4b9/0x700 [mlx4_core]
       local_pci_probe+0x42/0xa0
       local_pci_probe+0x42/0xa0
       work_for_cpu_fn+0x14/0x20
       work_for_cpu_fn+0x14/0x20
       process_one_work+0x1fd/0x630
       process_one_work+0x1fd/0x630
       worker_thread+0x1db/0x3b0
       worker_thread+0x1db/0x3b0
       kthread+0x11e/0x150
       kthread+0x11e/0x150
       ret_from_fork+0x27/0x40
       ret_from_fork+0x27/0x40

-> #1 ((&wfc.work)){+.+.}:

-> #1 ((&wfc.work)){+.+.}:
       lock_acquire+0xdc/0x1d0
       lock_acquire+0xdc/0x1d0
       process_one_work+0x1da/0x630
       process_one_work+0x1da/0x630
       worker_thread+0x4e/0x3b0
       worker_thread+0x4e/0x3b0
       kthread+0x11e/0x150
       kthread+0x11e/0x150
       ret_from_fork+0x27/0x40
       ret_from_fork+0x27/0x40

-> #0 ("events"){+.+.}:

-> #0 ("events"){+.+.}:
       __lock_acquire+0x13b5/0x13f0
       __lock_acquire+0x13b5/0x13f0
       lock_acquire+0xdc/0x1d0
       lock_acquire+0xdc/0x1d0
       flush_workqueue+0x98/0x520
       flush_workqueue+0x98/0x520
       nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
       nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
       ib_unregister_client+0x18f/0x200 [ib_core]
       ib_unregister_client+0x18f/0x200 [ib_core]
       nvmet_rdma_exit+0xb3/0x856 [nvmet_rdma]
       nvmet_rdma_exit+0xb3/0x856 [nvmet_rdma]
       SyS_delete_module+0x18c/0x1e0
       SyS_delete_module+0x18c/0x1e0
       entry_SYSCALL_64_fastpath+0x18/0xad
       entry_SYSCALL_64_fastpath+0x18/0xad

other info that might help us debug this:


other info that might help us debug this:

Chain exists of:
  "events" --> intf_mutex --> device_mutex

Chain exists of:
  "events" --> intf_mutex --> device_mutex

 Possible unsafe locking scenario:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       CPU0                    CPU1
       ----                    ----
       ----                    ----
  lock(device_mutex);
  lock(device_mutex);
                               lock(intf_mutex);
                               lock(intf_mutex);
                               lock(device_mutex);
                               lock(device_mutex);
  lock("events");
  lock("events");

 *** DEADLOCK ***


 *** DEADLOCK ***

1 lock held by modprobe/2272:
1 lock held by modprobe/2272:
 #0:  (device_mutex){+.+.}, at: [<ffffffffa05d6bf7>] ib_unregister_client+0x27/0x200 [ib_core]
 #0:  (device_mutex){+.+.}, at: [<ffffffffa05d6bf7>] ib_unregister_client+0x27/0x200 [ib_core]

stack backtrace:

stack backtrace:
CPU: 9 PID: 2272 Comm: modprobe Not tainted 4.14.0-rc5-dbg+ #3
CPU: 9 PID: 2272 Comm: modprobe Not tainted 4.14.0-rc5-dbg+ #3
Hardware name: Dell Inc. PowerEdge R720/0VWT90, BIOS 1.3.6 09/11/2012
Hardware name: Dell Inc. PowerEdge R720/0VWT90, BIOS 1.3.6 09/11/2012
Call Trace:
Call Trace:
 dump_stack+0x68/0x9f
 dump_stack+0x68/0x9f
 print_circular_bug.isra.38+0x1d8/0x1e6
 print_circular_bug.isra.38+0x1d8/0x1e6
 __lock_acquire+0x13b5/0x13f0
 __lock_acquire+0x13b5/0x13f0
 lock_acquire+0xdc/0x1d0
 lock_acquire+0xdc/0x1d0
 ? lock_acquire+0xdc/0x1d0
 ? lock_acquire+0xdc/0x1d0
 ? flush_workqueue+0x75/0x520
 ? flush_workqueue+0x75/0x520
 flush_workqueue+0x98/0x520
 flush_workqueue+0x98/0x520
 ? flush_workqueue+0x75/0x520
 ? flush_workqueue+0x75/0x520
 nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
 nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
 ? nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
 ? nvmet_rdma_remove_one+0x73/0xa0 [nvmet_rdma]
 ib_unregister_client+0x18f/0x200 [ib_core]
 ib_unregister_client+0x18f/0x200 [ib_core]
 nvmet_rdma_exit+0xb3/0x856 [nvmet_rdma]
 nvmet_rdma_exit+0xb3/0x856 [nvmet_rdma]
 SyS_delete_module+0x18c/0x1e0
 SyS_delete_module+0x18c/0x1e0
 entry_SYSCALL_64_fastpath+0x18/0xad
 entry_SYSCALL_64_fastpath+0x18/0xad

From ckulkarnilinux at gmail.com  Mon Oct 16 18:24:20 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Mon, 16 Oct 2017 18:24:20 -0700
Subject: [PATCH V7] nvme-pci: add SGL support
Message-ID: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds SGL support for NVMe PCIe driver, based on an earlier patch
from Rajiv Shanmugam Madeswaran <smrajiv15 at gmail.com>. This patch
refactors the original code and adds new module parameter sgl_threshold
to determine whether to use SGL or PRP for IOs.

The usage of SGLs is controlled by the sgl_threshold module parameter,
which allows to conditionally use SGLs if average request segment
size (avg_seg_size) is greater than sgl_threshold. In the original patch,
the decision of using SGLs was dependent only on the IO size,
with the new approach we consider not only IO size but also the
number of physical segments present in the IO.

We calculate avg_seg_size based on request payload bytes and number
of physical segments present in the request.

For e.g.:-

1. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 8k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

2. blk_rq_nr_phys_segments = 2 blk_rq_payload_bytes = 64k
avg_seg_size = 32K use sgl if avg_seg_size >= sgl_threshold.

3. blk_rq_nr_phys_segments = 16 blk_rq_payload_bytes = 64k
avg_seg_size = 4K use sgl if avg_seg_size >= sgl_threshold.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
This is the seventh version of the patch for adding SGL support
for nvme-pci driver.

Changes since v6:-

1. Refactor the code in nvme_setup_sgls() and get rid of
   the special case of handling single segment.
2. In the process of adding SGL support, we changed the signature
   of some functions. Add prefix "nvme_pci_" to those function.
3. Fix a typo in the commit message.
4. Add information about how to disable the SGLs in the
   description of newly added module parameter sgl_threshold.
5. Change the code base from Linux 4.14-rc3 to 4.14.0-rc4.

Changes since v5:-

1. Remove the NVME_CTRL_SGL_NOT_SUPP enum constant and use
   simple comparison bits to determine whether controller supports
   SGL or not.

Changes since v4:-

1. Replace NVME_CTRL_SUPP_SGL macro with enum NVME_CTRL_SGL_NOT_SUPP.
2. Adjust patch for new enum.
3. Merge two patches into one.

Changes since v3:-

1. Modify SGL helper functions and add prefix nvme_pci_sgl.
2. Use average request physical segment(s) size to conditionally use SGLs.
   This approach is different from the original one, the original approach
   was based no the IO size. The new approach considers IO size and
   number of physical segments present in the request to conditionally
   use SGLs.
3. Adjust the patch for Linux 4.14-rc3.

Changes since v2:-

1. Adjust the patch for latest nvme driver changes.
2. For initial submission use the default virt boundary values.

In future, work on the patch series for relaxing the virt boundary
for each transport.

Changes since the v1:-

1. Introduced nvme controller virt_boundary mask. For controllers that
   can handle arbitrarily sized bios (e.g advanced RDMA and PCI ctrls)
   we can allow the block layer to pass us gaps by setting the
   queue.virt_boundary_mask according to each ctrl constraints.
2. Simplified code for setting up the SGLs.
3. Added SGL helper functions.
4. Modified commit log.
5. Use the newly introduced virt_boundary mask and clear its
   value for SGL capable controllers in PCIe driver.

For reference v1 details:-

This adds SGL support for NVMe PCIe driver which is
reimplementation of the initial version provided by
Rajiv Shanmugam Madeswaran(smrajiv15 at gmail.com):-
"block: nvme-core: Scatter gather list support in the
NVMe Block Driver".

In order to enable SGL mode for the driver, the user can set the
sgl_threshold module parameter. This parameter can be used
in the two modes:-

1. Allow all IOs to use SGL mode. (set sgl_threshold to 1).
2. Set up the IO size threshold to determine whether to use
    SGLs or PRPs for each IO. (set sgl_threshold to 4096 to
    use SGL only for IOs which are >= 4096 in the size).
---
 drivers/nvme/host/pci.c | 214 ++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 187 insertions(+), 27 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3f5a04c..45a3397 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -45,6 +45,8 @@
  */
 #define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
 
+#define SGES_PER_PAGE	(PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+
 static int use_threaded_interrupts;
 module_param(use_threaded_interrupts, int, 0);
 
@@ -57,6 +59,12 @@
 MODULE_PARM_DESC(max_host_mem_size_mb,
 	"Maximum Host Memory Buffer (HMB) size per controller (in MiB)");
 
+static unsigned int sgl_threshold = SZ_32K;
+module_param(sgl_threshold, uint, 0644);
+MODULE_PARM_DESC(sgl_threshold,
+		"Use SGLs when average request segment size is larger or equal to "
+		"this size. Use 0 to disable SGLs.");
+
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
 static const struct kernel_param_ops io_queue_depth_ops = {
 	.set = io_queue_depth_set,
@@ -178,6 +186,7 @@ struct nvme_queue {
 struct nvme_iod {
 	struct nvme_request req;
 	struct nvme_queue *nvmeq;
+	bool use_sgl;
 	int aborted;
 	int npages;		/* In the PRP list. 0 means small pool in use */
 	int nents;		/* Used in scatterlist */
@@ -331,17 +340,35 @@ static int nvme_npages(unsigned size, struct nvme_dev *dev)
 	return DIV_ROUND_UP(8 * nprps, PAGE_SIZE - 8);
 }
 
-static unsigned int nvme_iod_alloc_size(struct nvme_dev *dev,
-		unsigned int size, unsigned int nseg)
+/*
+ * Calculates the number of pages needed for the SGL segments. For example a 4k
+ * page can accommodate 256 SGL descriptors.
+ */
+static int nvme_pci_npages_sgl(unsigned int num_seg)
+{
+	return DIV_ROUND_UP(num_seg * sizeof(struct nvme_sgl_desc), PAGE_SIZE);
+}
+
+static unsigned int nvme_pci_iod_alloc_size(struct nvme_dev *dev,
+		unsigned int size, unsigned int nseg, bool use_sgl)
 {
-	return sizeof(__le64 *) * nvme_npages(size, dev) +
-			sizeof(struct scatterlist) * nseg;
+	size_t alloc_size;
+
+	if (use_sgl)
+		alloc_size = sizeof(__le64 *) * nvme_pci_npages_sgl(nseg);
+	else
+		alloc_size = sizeof(__le64 *) * nvme_npages(size, dev);
+
+	return alloc_size + sizeof(struct scatterlist) * nseg;
 }
 
-static unsigned int nvme_cmd_size(struct nvme_dev *dev)
+static unsigned int nvme_pci_cmd_size(struct nvme_dev *dev, bool use_sgl)
 {
-	return sizeof(struct nvme_iod) +
-		nvme_iod_alloc_size(dev, NVME_INT_BYTES(dev), NVME_INT_PAGES);
+	unsigned int alloc_size = nvme_pci_iod_alloc_size(dev,
+				    NVME_INT_BYTES(dev), NVME_INT_PAGES,
+				    use_sgl);
+
+	return sizeof(struct nvme_iod) + alloc_size;
 }
 
 static int nvme_admin_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
@@ -425,10 +452,10 @@ static void __nvme_submit_cmd(struct nvme_queue *nvmeq,
 	nvmeq->sq_tail = tail;
 }
 
-static __le64 **iod_list(struct request *req)
+static void **nvme_pci_iod_list(struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	return (__le64 **)(iod->sg + blk_rq_nr_phys_segments(req));
+	return (void **)(iod->sg + blk_rq_nr_phys_segments(req));
 }
 
 static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
@@ -438,7 +465,10 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 	unsigned int size = blk_rq_payload_bytes(rq);
 
 	if (nseg > NVME_INT_PAGES || size > NVME_INT_BYTES(dev)) {
-		iod->sg = kmalloc(nvme_iod_alloc_size(dev, size, nseg), GFP_ATOMIC);
+		size_t alloc_size = nvme_pci_iod_alloc_size(dev, size, nseg,
+				iod->use_sgl);
+
+		iod->sg = kmalloc(alloc_size, GFP_ATOMIC);
 		if (!iod->sg)
 			return BLK_STS_RESOURCE;
 	} else {
@@ -456,18 +486,31 @@ static blk_status_t nvme_init_iod(struct request *rq, struct nvme_dev *dev)
 static void nvme_free_iod(struct nvme_dev *dev, struct request *req)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
-	const int last_prp = dev->ctrl.page_size / 8 - 1;
+	const int last_prp = dev->ctrl.page_size / sizeof(__le64) - 1;
+	dma_addr_t dma_addr = iod->first_dma, next_dma_addr;
+
 	int i;
-	__le64 **list = iod_list(req);
-	dma_addr_t prp_dma = iod->first_dma;
 
 	if (iod->npages == 0)
-		dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
+		dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0],
+			dma_addr);
+
 	for (i = 0; i < iod->npages; i++) {
-		__le64 *prp_list = list[i];
-		dma_addr_t next_prp_dma = le64_to_cpu(prp_list[last_prp]);
-		dma_pool_free(dev->prp_page_pool, prp_list, prp_dma);
-		prp_dma = next_prp_dma;
+		void *addr = nvme_pci_iod_list(req)[i];
+
+		if (iod->use_sgl) {
+			struct nvme_sgl_desc *sg_list = addr;
+
+			next_dma_addr =
+			    le64_to_cpu((sg_list[SGES_PER_PAGE - 1]).addr);
+		} else {
+			__le64 *prp_list = addr;
+
+			next_dma_addr = le64_to_cpu(prp_list[last_prp]);
+		}
+
+		dma_pool_free(dev->prp_page_pool, addr, dma_addr);
+		dma_addr = next_dma_addr;
 	}
 
 	if (iod->sg != iod->inline_sg)
@@ -555,7 +598,8 @@ static void nvme_print_sgl(struct scatterlist *sgl, int nents)
 	}
 }
 
-static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
+static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev,
+		struct request *req, struct nvme_rw_command *cmnd)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
 	struct dma_pool *pool;
@@ -566,14 +610,16 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	u32 page_size = dev->ctrl.page_size;
 	int offset = dma_addr & (page_size - 1);
 	__le64 *prp_list;
-	__le64 **list = iod_list(req);
+	void **list = nvme_pci_iod_list(req);
 	dma_addr_t prp_dma;
 	int nprps, i;
 
+	iod->use_sgl = false;
+
 	length -= (page_size - offset);
 	if (length <= 0) {
 		iod->first_dma = 0;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	dma_len -= (page_size - offset);
@@ -587,7 +633,7 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 
 	if (length <= page_size) {
 		iod->first_dma = dma_addr;
-		return BLK_STS_OK;
+		goto done;
 	}
 
 	nprps = DIV_ROUND_UP(length, page_size);
@@ -634,6 +680,10 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 		dma_len = sg_dma_len(sg);
 	}
 
+done:
+	cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
+	cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma);
+
 	return BLK_STS_OK;
 
  bad_sgl:
@@ -643,6 +693,110 @@ static blk_status_t nvme_setup_prps(struct nvme_dev *dev, struct request *req)
 	return BLK_STS_IOERR;
 }
 
+static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
+		struct scatterlist *sg)
+{
+	sge->addr = cpu_to_le64(sg_dma_address(sg));
+	sge->length = cpu_to_le32(sg_dma_len(sg));
+	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
+}
+
+static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
+		dma_addr_t dma_addr, int entries)
+{
+	sge->addr = cpu_to_le64(dma_addr);
+	if (entries < SGES_PER_PAGE) {
+		sge->length = cpu_to_le32(entries * sizeof(*sge));
+		sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
+	} else {
+		sge->length = cpu_to_le32(PAGE_SIZE);
+		sge->type = NVME_SGL_FMT_SEG_DESC << 4;
+	}
+}
+
+static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev,
+		struct request *req, struct nvme_rw_command *cmd)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	int length = blk_rq_payload_bytes(req);
+	struct dma_pool *pool;
+	struct nvme_sgl_desc *sg_list;
+	struct scatterlist *sg = iod->sg;
+	int entries = iod->nents, i = 0;
+	dma_addr_t sgl_dma;
+
+	iod->use_sgl = true;
+
+	/* setting the transfer type as SGL */
+	cmd->flags = NVME_CMD_SGL_METABUF;
+
+	if (length == sg_dma_len(sg)) {
+		nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg);
+		return BLK_STS_OK;
+	}
+
+	if (entries <= (256 / sizeof(struct nvme_sgl_desc))) {
+		pool = dev->prp_small_pool;
+		iod->npages = 0;
+	} else {
+		pool = dev->prp_page_pool;
+		iod->npages = 1;
+	}
+
+	sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+	if (!sg_list) {
+		iod->npages = -1;
+		return BLK_STS_RESOURCE;
+	}
+
+	nvme_pci_iod_list(req)[0] = sg_list;
+	iod->first_dma = sgl_dma;
+
+	nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries);
+
+	do {
+		if (i == SGES_PER_PAGE) {
+			struct nvme_sgl_desc *old_sg_desc = sg_list;
+			struct nvme_sgl_desc *link = &old_sg_desc[i - 1];
+
+			sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma);
+			if (!sg_list)
+				return BLK_STS_RESOURCE;
+
+			i = 0;
+			nvme_pci_iod_list(req)[iod->npages++] = sg_list;
+			sg_list[i++] = *link;
+			nvme_pci_sgl_set_seg(link, sgl_dma, entries);
+		}
+
+		nvme_pci_sgl_set_data(&sg_list[i++], sg);
+
+		length -= sg_dma_len(sg);
+		sg = sg_next(sg);
+		entries--;
+	} while (length > 0);
+
+	WARN_ON(entries > 0);
+	return BLK_STS_OK;
+}
+
+static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	unsigned int avg_seg_size;
+
+	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req),
+			blk_rq_nr_phys_segments(req));
+
+	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
+		return false;
+	if (!iod->nvmeq->qid)
+		return false;
+	if (!sgl_threshold || avg_seg_size < sgl_threshold)
+		return false;
+	return true;
+}
+
 static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 		struct nvme_command *cmnd)
 {
@@ -662,7 +816,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 				DMA_ATTR_NO_WARN))
 		goto out;
 
-	ret = nvme_setup_prps(dev, req);
+	if (nvme_pci_use_sgls(dev, req))
+		ret = nvme_pci_setup_sgls(dev, req, &cmnd->rw);
+	else
+		ret = nvme_pci_setup_prps(dev, req, &cmnd->rw);
+
 	if (ret != BLK_STS_OK)
 		goto out_unmap;
 
@@ -682,8 +840,6 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 			goto out_unmap;
 	}
 
-	cmnd->rw.dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg));
-	cmnd->rw.dptr.prp2 = cpu_to_le64(iod->first_dma);
 	if (blk_integrity_rq(req))
 		cmnd->rw.metadata = cpu_to_le64(sg_dma_address(&iod->meta_sg));
 	return BLK_STS_OK;
@@ -1379,7 +1535,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
-		dev->admin_tagset.cmd_size = nvme_cmd_size(dev);
+		dev->admin_tagset.cmd_size = nvme_pci_cmd_size(dev, false);
 		dev->admin_tagset.flags = BLK_MQ_F_NO_SCHED;
 		dev->admin_tagset.driver_data = dev;
 
@@ -1906,7 +2062,11 @@ static int nvme_dev_add(struct nvme_dev *dev)
 		dev->tagset.numa_node = dev_to_node(dev->dev);
 		dev->tagset.queue_depth =
 				min_t(int, dev->q_depth, BLK_MQ_MAX_DEPTH) - 1;
-		dev->tagset.cmd_size = nvme_cmd_size(dev);
+		dev->tagset.cmd_size = nvme_pci_cmd_size(dev, false);
+		if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
+			dev->tagset.cmd_size = max(dev->tagset.cmd_size,
+					nvme_pci_cmd_size(dev, true));
+		}
 		dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE;
 		dev->tagset.driver_data = dev;
 
-- 
1.8.3.1



From hare at suse.de  Mon Oct 16 23:06:42 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:06:42 +0200
Subject: [PATCH v2 1/8] lib/scatterlist: Introduce sgl_alloc() and
 sgl_free()
In-Reply-To: <20171016224940.1332-2-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-2-bart.vanassche@wdc.com>
Message-ID: <b7975651-7f1e-9a61-f752-78eb8763dece@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Many kernel drivers contain code that allocates and frees both a
> scatterlist and the pages that populate that scatterlist.
> Introduce functions in lib/scatterlist.c that perform these tasks
> instead of duplicating this functionality in multiple drivers.
> Only include these functions in the build if CONFIG_SGL_ALLOC=y
> to avoid that the kernel size increases if this functionality is
> not used.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> ---
>  include/linux/scatterlist.h |  10 +++++
>  lib/Kconfig                 |   4 ++
>  lib/scatterlist.c           | 105 ++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 119 insertions(+)
> 

Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:07:20 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:07:20 +0200
Subject: [PATCH v2 2/8] crypto: scompress - use sgl_alloc() and sgl_free()
In-Reply-To: <20171016224940.1332-3-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-3-bart.vanassche@wdc.com>
Message-ID: <9e6d6115-8118-4e8d-aacb-849598becd19@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc() and sgl_free() functions instead of open coding
> these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Cc: Ard Biesheuvel <ard.biesheuvel at linaro.org>
> Cc: Herbert Xu <herbert at gondor.apana.org.au>
> ---
>  crypto/Kconfig     |  1 +
>  crypto/scompress.c | 51 ++-------------------------------------------------
>  2 files changed, 3 insertions(+), 49 deletions(-)
> Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:07:48 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:07:48 +0200
Subject: [PATCH v2 3/8] nvmet/fc: Use sgl_alloc() and sgl_free()
In-Reply-To: <20171016224940.1332-4-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-4-bart.vanassche@wdc.com>
Message-ID: <97bcefd4-bff2-9b14-7981-21e238c2d2e6@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc() and sgl_free() functions instead of open coding
> these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: Keith Busch <keith.busch at intel.com>
> Cc: Christoph Hellwig <hch at lst.de>
> Cc: James Smart <james.smart at broadcom.com>
> Cc: Sagi Grimberg <sagi at grimberg.me>
> ---
>  drivers/nvme/target/Kconfig |  1 +
>  drivers/nvme/target/fc.c    | 36 ++----------------------------------
>  2 files changed, 3 insertions(+), 34 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:08:23 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:08:23 +0200
Subject: [PATCH v2 4/8] nvmet/rdma: Use sgl_alloc() and sgl_free()
In-Reply-To: <20171016224940.1332-5-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-5-bart.vanassche@wdc.com>
Message-ID: <3169fd91-8a1c-5b9f-cdb0-2ffafe25f41e@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc() and sgl_free() functions instead of open coding
> these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: Keith Busch <keith.busch at intel.com>
> Cc: Christoph Hellwig <hch at lst.de>
> Cc: Sagi Grimberg <sagi at grimberg.me>
> ---
>  drivers/nvme/target/Kconfig |  1 +
>  drivers/nvme/target/rdma.c  | 63 +++------------------------------------------
>  2 files changed, 5 insertions(+), 59 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:14:10 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:14:10 +0200
Subject: [PATCH v2 5/8] target: Use sgl_alloc_order() and sgl_free()
In-Reply-To: <20171016224940.1332-6-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-6-bart.vanassche@wdc.com>
Message-ID: <c5e05f84-2602-5c3f-521e-6f35601011cf@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc_order() and sgl_free() functions instead of open
> coding these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Cc: Nicholas A. Bellinger <nab at linux-iscsi.org>
> Cc: Christoph Hellwig <hch at lst.de>
> Cc: Hannes Reinecke <hare at suse.com>
> Cc: Sagi Grimberg <sagi at grimberg.me>
> ---
>  drivers/target/Kconfig                 |  1 +
>  drivers/target/target_core_transport.c | 46 +++-------------------------------
>  2 files changed, 5 insertions(+), 42 deletions(-)
> 
> diff --git a/drivers/target/Kconfig b/drivers/target/Kconfig
> index e2bc99980f75..4c44d7bed01a 100644
> --- a/drivers/target/Kconfig
> +++ b/drivers/target/Kconfig
> @@ -5,6 +5,7 @@ menuconfig TARGET_CORE
>  	select CONFIGFS_FS
>  	select CRC_T10DIF
>  	select BLK_SCSI_REQUEST # only for scsi_command_size_tbl..
> +	select SGL_ALLOC
>  	default n
>  	help
>  	Say Y or M here to enable the TCM Storage Engine and ConfigFS enabled
> diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
> index 836d552b0385..9bbd08be9d60 100644
> --- a/drivers/target/target_core_transport.c
> +++ b/drivers/target/target_core_transport.c
> @@ -2293,13 +2293,7 @@ static void target_complete_ok_work(struct work_struct *work)
>  
>  void target_free_sgl(struct scatterlist *sgl, int nents)
>  {
> -	struct scatterlist *sg;
> -	int count;
> -
> -	for_each_sg(sgl, sg, nents, count)
> -		__free_page(sg_page(sg));
> -
> -	kfree(sgl);
> +	sgl_free(sgl);
>  }
>  EXPORT_SYMBOL(target_free_sgl);
>  
> @@ -2423,42 +2417,10 @@ int
>  target_alloc_sgl(struct scatterlist **sgl, unsigned int *nents, u32 length,
>  		 bool zero_page, bool chainable)
>  {
> -	struct scatterlist *sg;
> -	struct page *page;
> -	gfp_t zero_flag = (zero_page) ? __GFP_ZERO : 0;
> -	unsigned int nalloc, nent;
> -	int i = 0;
> -
> -	nalloc = nent = DIV_ROUND_UP(length, PAGE_SIZE);
> -	if (chainable)
> -		nalloc++;
> -	sg = kmalloc_array(nalloc, sizeof(struct scatterlist), GFP_KERNEL);
> -	if (!sg)
> -		return -ENOMEM;
> +	gfp_t gfp = GFP_KERNEL | (zero_page ? __GFP_ZERO : 0);
>  
> -	sg_init_table(sg, nalloc);
> -
> -	while (length) {
> -		u32 page_len = min_t(u32, length, PAGE_SIZE);
> -		page = alloc_page(GFP_KERNEL | zero_flag);
> -		if (!page)
> -			goto out;
> -
> -		sg_set_page(&sg[i], page, page_len, 0);
> -		length -= page_len;
> -		i++;
> -	}
> -	*sgl = sg;
> -	*nents = nent;
> -	return 0;
> -
> -out:
> -	while (i > 0) {
> -		i--;
> -		__free_page(sg_page(&sg[i]));
> -	}
> -	kfree(sg);
> -	return -ENOMEM;
> +	*sgl = sgl_alloc_order(length, 0, chainable, gfp, nents);
> +	return *sgl ? 0 : -ENOMEM;
>  }
>  EXPORT_SYMBOL(target_alloc_sgl);
>  
> The calling convention from target_alloc_sgl() is decidedly dodgy.
Can't we convert it into returning the sgl, and remove the first parameter?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:19:05 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:19:05 +0200
Subject: [PATCH v2 6/8] scsi/ipr: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <20171016224940.1332-7-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-7-bart.vanassche@wdc.com>
Message-ID: <c804098f-5543-395c-1395-5778ef74a61b@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc_order() and sgl_free_order() functions instead
> of open coding these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: linux-scsi at vger.kernel.org
> Cc: Martin K. Petersen <martin.petersen at oracle.com>
> Cc: Brian King <brking at linux.vnet.ibm.com>
> ---
>  drivers/scsi/Kconfig |  1 +
>  drivers/scsi/ipr.c   | 49 ++++++++-----------------------------------------
>  drivers/scsi/ipr.h   |  2 +-
>  3 files changed, 10 insertions(+), 42 deletions(-)
> 
> diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
> index 41366339b950..d11e75e76195 100644
> --- a/drivers/scsi/Kconfig
> +++ b/drivers/scsi/Kconfig
> @@ -1058,6 +1058,7 @@ config SCSI_IPR
>  	depends on PCI && SCSI && ATA
>  	select FW_LOADER
>  	select IRQ_POLL
> +	select SGL_ALLOC
>  	---help---
>  	  This driver supports the IBM Power Linux family RAID adapters.
>  	  This includes IBM pSeries 5712, 5703, 5709, and 570A, as well
> diff --git a/drivers/scsi/ipr.c b/drivers/scsi/ipr.c
> index f838bd73befa..6fed5db6255e 100644
> --- a/drivers/scsi/ipr.c
> +++ b/drivers/scsi/ipr.c
> @@ -3815,10 +3815,8 @@ static struct device_attribute ipr_iopoll_weight_attr = {
>   **/
>  static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>  {
> -	int sg_size, order, bsize_elem, num_elem, i, j;
> +	int sg_size, order;
>  	struct ipr_sglist *sglist;
> -	struct scatterlist *scatterlist;
> -	struct page *page;
>  
>  	/* Get the minimum size per scatter/gather element */
>  	sg_size = buf_len / (IPR_MAX_SGLIST - 1);
> @@ -3826,45 +3824,18 @@ static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>  	/* Get the actual size per element */
>  	order = get_order(sg_size);
>  
> -	/* Determine the actual number of bytes per element */
> -	bsize_elem = PAGE_SIZE * (1 << order);
> -
> -	/* Determine the actual number of sg entries needed */
> -	if (buf_len % bsize_elem)
> -		num_elem = (buf_len / bsize_elem) + 1;
> -	else
> -		num_elem = buf_len / bsize_elem;
> -
>  	/* Allocate a scatter/gather list for the DMA */
> -	sglist = kzalloc(sizeof(struct ipr_sglist) +
> -			 (sizeof(struct scatterlist) * (num_elem - 1)),
> -			 GFP_KERNEL);
> -
> +	sglist = kzalloc(sizeof(struct ipr_sglist), GFP_KERNEL);
>  	if (sglist == NULL) {
>  		ipr_trace;
>  		return NULL;
>  	}
> -
> -	scatterlist = sglist->scatterlist;
> -	sg_init_table(scatterlist, num_elem);
> -
>  	sglist->order = order;
> -	sglist->num_sg = num_elem;
> -
> -	/* Allocate a bunch of sg elements */
> -	for (i = 0; i < num_elem; i++) {
> -		page = alloc_pages(GFP_KERNEL, order);
> -		if (!page) {
> -			ipr_trace;
> -
> -			/* Free up what we already allocated */
> -			for (j = i - 1; j >= 0; j--)
> -				__free_pages(sg_page(&scatterlist[j]), order);
> -			kfree(sglist);
> -			return NULL;
> -		}
> -
> -		sg_set_page(&scatterlist[i], page, 0, 0);
> +	sglist->scatterlist = sgl_alloc_order(buf_len, order, false, GFP_KERNEL,
> +					      &sglist->num_sg);
> +	if (!sglist->scatterlist) {
> +		kfree(sglist);
> +		return NULL;
>  	}
>  
>  	return sglist;
> @@ -3882,11 +3853,7 @@ static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>   **/
>  static void ipr_free_ucode_buffer(struct ipr_sglist *sglist)
>  {
> -	int i;
> -
> -	for (i = 0; i < sglist->num_sg; i++)
> -		__free_pages(sg_page(&sglist->scatterlist[i]), sglist->order);
> -
> +	sgl_free_order(sglist->scatterlist, sglist->order);
>  	kfree(sglist);
>  }
>  
> diff --git a/drivers/scsi/ipr.h b/drivers/scsi/ipr.h
> index c7f0e9e3cd7d..93570734cbfb 100644
> --- a/drivers/scsi/ipr.h
> +++ b/drivers/scsi/ipr.h
> @@ -1454,7 +1454,7 @@ struct ipr_sglist {
>  	u32 num_sg;
>  	u32 num_dma_sg;
>  	u32 buffer_len;
> -	struct scatterlist scatterlist[1];
> +	struct scatterlist *scatterlist;
>  };
>  
>  enum ipr_sdt_state {
> 
Not sure if this is a valid conversion.
Originally the driver would allocate a single buffer; with this buffer
we have two distinct buffers.
Given that this is used to download the microcode I'm not sure if this
isn't a hardware-dependent structure which requires a single buffer
including the sglist.
Brian, can you shed some light here?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:21:33 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:21:33 +0200
Subject: [PATCH v2 7/8] scsi/pmcraid: Remove an unused structure member
In-Reply-To: <20171016224940.1332-8-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-8-bart.vanassche@wdc.com>
Message-ID: <5ea64c10-aa3e-f0ee-cfba-e93b3d8e1428@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: linux-scsi at vger.kernel.org
> Cc: Martin K. Petersen <martin.petersen at oracle.com>
> Cc: Anil Ravindranath <anil_ravindranath at pmc-sierra.com>
> ---
>  drivers/scsi/pmcraid.h | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/scsi/pmcraid.h b/drivers/scsi/pmcraid.h
> index 8bfac72a242b..44da91712115 100644
> --- a/drivers/scsi/pmcraid.h
> +++ b/drivers/scsi/pmcraid.h
> @@ -542,7 +542,6 @@ struct pmcraid_sglist {
>  	u32 order;
>  	u32 num_sg;
>  	u32 num_dma_sg;
> -	u32 buffer_len;
>  	struct scatterlist scatterlist[1];
>  };
>  
> 
This actually is the same story that we've had with ipr (and, looking at
the code, those two drivers look awfully similar ...).
pmcraid_sglist looks as if it's a hardware-dependent structure, so just
removing one entry from the middle of a structure might not be a good idea.
But this is something for the pmcraid folks to clarify.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:22:18 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:22:18 +0200
Subject: [PATCH v2 8/8] scsi/pmcraid: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <20171016224940.1332-9-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-9-bart.vanassche@wdc.com>
Message-ID: <00b49c32-505e-410a-40dd-9d6f793354be@suse.de>

On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> Use the sgl_alloc_order() and sgl_free_order() functions instead
> of open coding these functions.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: linux-scsi at vger.kernel.org
> Cc: Martin K. Petersen <martin.petersen at oracle.com>
> Cc: Anil Ravindranath <anil_ravindranath at pmc-sierra.com>
> ---
>  drivers/scsi/Kconfig   |  1 +
>  drivers/scsi/pmcraid.c | 43 ++++---------------------------------------
>  drivers/scsi/pmcraid.h |  2 +-
>  3 files changed, 6 insertions(+), 40 deletions(-)
> 
> diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
> index d11e75e76195..632200ec36a6 100644
> --- a/drivers/scsi/Kconfig
> +++ b/drivers/scsi/Kconfig
> @@ -1576,6 +1576,7 @@ config ZFCP
>  config SCSI_PMCRAID
>  	tristate "PMC SIERRA Linux MaxRAID adapter support"
>  	depends on PCI && SCSI && NET
> +	select SGL_ALLOC
>  	---help---
>  	  This driver supports the PMC SIERRA MaxRAID adapters.
>  
> diff --git a/drivers/scsi/pmcraid.c b/drivers/scsi/pmcraid.c
> index b4d6cd8cd1ad..95fc0352f9bb 100644
> --- a/drivers/scsi/pmcraid.c
> +++ b/drivers/scsi/pmcraid.c
> @@ -3232,12 +3232,7 @@ static int pmcraid_build_ioadl(
>   */
>  static void pmcraid_free_sglist(struct pmcraid_sglist *sglist)
>  {
> -	int i;
> -
> -	for (i = 0; i < sglist->num_sg; i++)
> -		__free_pages(sg_page(&(sglist->scatterlist[i])),
> -			     sglist->order);
> -
> +	sgl_free_order(sglist->scatterlist, sglist->order);
>  	kfree(sglist);
>  }
>  
> @@ -3254,50 +3249,20 @@ static void pmcraid_free_sglist(struct pmcraid_sglist *sglist)
>  static struct pmcraid_sglist *pmcraid_alloc_sglist(int buflen)
>  {
>  	struct pmcraid_sglist *sglist;
> -	struct scatterlist *scatterlist;
> -	struct page *page;
> -	int num_elem, i, j;
>  	int sg_size;
>  	int order;
> -	int bsize_elem;
>  
>  	sg_size = buflen / (PMCRAID_MAX_IOADLS - 1);
>  	order = (sg_size > 0) ? get_order(sg_size) : 0;
> -	bsize_elem = PAGE_SIZE * (1 << order);
> -
> -	/* Determine the actual number of sg entries needed */
> -	if (buflen % bsize_elem)
> -		num_elem = (buflen / bsize_elem) + 1;
> -	else
> -		num_elem = buflen / bsize_elem;
>  
>  	/* Allocate a scatter/gather list for the DMA */
> -	sglist = kzalloc(sizeof(struct pmcraid_sglist) +
> -			 (sizeof(struct scatterlist) * (num_elem - 1)),
> -			 GFP_KERNEL);
> -
> +	sglist = kzalloc(sizeof(struct pmcraid_sglist), GFP_KERNEL);
>  	if (sglist == NULL)
>  		return NULL;
>  
> -	scatterlist = sglist->scatterlist;
> -	sg_init_table(scatterlist, num_elem);
>  	sglist->order = order;
> -	sglist->num_sg = num_elem;
> -	sg_size = buflen;
> -
> -	for (i = 0; i < num_elem; i++) {
> -		page = alloc_pages(GFP_KERNEL|GFP_DMA|__GFP_ZERO, order);
> -		if (!page) {
> -			for (j = i - 1; j >= 0; j--)
> -				__free_pages(sg_page(&scatterlist[j]), order);
> -			kfree(sglist);
> -			return NULL;
> -		}
> -
> -		sg_set_page(&scatterlist[i], page,
> -			sg_size < bsize_elem ? sg_size : bsize_elem, 0);
> -		sg_size -= bsize_elem;
> -	}
> +	sgl_alloc_order(buflen, order, false,
> +			GFP_KERNEL | GFP_DMA | __GFP_ZERO, &sglist->num_sg);
>  
>  	return sglist;
>  }
> diff --git a/drivers/scsi/pmcraid.h b/drivers/scsi/pmcraid.h
> index 44da91712115..754ef30927e2 100644
> --- a/drivers/scsi/pmcraid.h
> +++ b/drivers/scsi/pmcraid.h
> @@ -542,7 +542,7 @@ struct pmcraid_sglist {
>  	u32 order;
>  	u32 num_sg;
>  	u32 num_dma_sg;
> -	struct scatterlist scatterlist[1];
> +	struct scatterlist *scatterlist;
>  };
>  
>  /* page D0 inquiry data of focal point resource */
> 
The comment to ipr applied here, too.
We need input from the pmcraid folks if this is a valid conversion.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Mon Oct 16 23:34:19 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 17 Oct 2017 08:34:19 +0200
Subject: [PATCH v9 10/10] block, nvme: Introduce blk_mq_req_flags_t
In-Reply-To: <20171016232905.5047-11-bart.vanassche@wdc.com>
References: <20171016232905.5047-1-bart.vanassche@wdc.com>
 <20171016232905.5047-11-bart.vanassche@wdc.com>
Message-ID: <56950f16-7c51-3e58-edd8-2cdf9049f686@suse.de>

On 10/17/2017 01:29 AM, Bart Van Assche wrote:
> Several block layer and NVMe core functions accept a combination
> of BLK_MQ_REQ_* flags through the 'flags' argument but there is
> no verification at compile time whether the right type of block
> layer flags is passed. Make it possible for sparse to verify this.
> This patch does not change any functionality.
> 
> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> Cc: linux-nvme at lists.infradead.org
> Cc: Christoph Hellwig <hch at lst.de>
> Cc: Hannes Reinecke <hare at suse.com>
> Cc: Johannes Thumshirn <jthumshirn at suse.de>
> Cc: Ming Lei <ming.lei at redhat.com>
> ---
>  block/blk-core.c          | 12 ++++++------
>  block/blk-mq.c            |  4 ++--
>  block/blk-mq.h            |  2 +-
>  drivers/nvme/host/core.c  |  5 +++--
>  drivers/nvme/host/nvme.h  |  5 +++--
>  include/linux/blk-mq.h    | 17 +++++++++++------
>  include/linux/blk_types.h |  2 ++
>  include/linux/blkdev.h    |  4 ++--
>  8 files changed, 30 insertions(+), 21 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hch at infradead.org  Mon Oct 16 23:43:06 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 23:43:06 -0700
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
In-Reply-To: <20171016151820.14766-1-jsmart2021@gmail.com>
References: <20171016151820.14766-1-jsmart2021@gmail.com>
Message-ID: <20171017064306.GA16133@infradead.org>

>  	if (status)
>  		nvmet_set_status(req, status);
>  
> +	spin_lock_irqsave(&req->sq->sqhd_lock, flags);
>  	if (req->sq->size)
>  		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
>  	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
> +	spin_unlock_irqrestore(&req->sq->sqhd_lock, flags);

What performance impact does this have?   I'm really reluctant to put
an irq disabling spinlock into a hot path for a feature that
theoretically is in the spec but ignored by every host.

I'd much rather play games with cmpxchg or similar here.


From hch at infradead.org  Mon Oct 16 23:50:55 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 23:50:55 -0700
Subject: [PATCH v2 0/3] Move tagset reinit to its only current consumer,
 nvme-core
In-Reply-To: <718a34b5-b28e-fc4f-5b42-f961dde53482@kernel.dk>
References: <1507715588-1041-1-git-send-email-sagi@grimberg.me>
 <20171016124817.GB26833@infradead.org>
 <718a34b5-b28e-fc4f-5b42-f961dde53482@kernel.dk>
Message-ID: <20171017065055.GA4164@infradead.org>

On Mon, Oct 16, 2017 at 08:47:38AM -0600, Jens Axboe wrote:
> On 10/16/2017 06:48 AM, Christoph Hellwig wrote:
> > Jens, are you fine with picking this up throught the nvme tree?
> > I think Sagi's later changes rely on it, so that would make life
> > a littler easier.
> 
> Yeah that's fine, you can add my reviewed-by to the patches
> as well.

Ok, applied to nvme-4.15 with your ACK.


From hch at infradead.org  Mon Oct 16 23:51:23 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 16 Oct 2017 23:51:23 -0700
Subject: [PATCH 0/7] Some nvme-rdma host patches for 4.15
In-Reply-To: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
References: <1507724952-13917-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171017065123.GB4164@infradead.org>

Applied to nvme-4.15 with the typo in patch 6 fixed up.


From jthumshirn at suse.de  Tue Oct 17 01:48:38 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Tue, 17 Oct 2017 10:48:38 +0200
Subject: [PATCH nvme-cli] fabrics: also set queue_size and nr_io_queues on
 connect-all connects
Message-ID: <20171017084838.6827-1-jthumshirn@suse.de>

Although nvme connect-all -Q does set the queue_size parameter for the initial
discovery connection, it does not set it for the subsequent connections to the
discovered targets.

An
yields the following:

write(3, "nqn=nqn.2014-08.org.nvmexpress.d"..., 180) = 180
 | 00000  6e 71 6e 3d 6e 71 6e 2e  32 30 31 34 2d 30 38 2e  nqn=nqn.2014-08. |
 | 00010  6f 72 67 2e 6e 76 6d 65  78 70 72 65 73 73 2e 64  org.nvmexpress.d |
 | 00020  69 73 63 6f 76 65 72 79  2c 74 72 61 6e 73 70 6f  iscovery,transpo |
 | 00030  72 74 3d 72 64 6d 61 2c  74 72 61 64 64 72 3d 31  rt=rdma,traddr=1 |
 | 00040  2e 31 2e 31 2e 33 2c 74  72 73 76 63 69 64 3d 34  .1.1.3,trsvcid=4 |
 | 00050  34 32 30 2c 68 6f 73 74  6e 71 6e 3d 6e 71 6e 2e  420,hostnqn=nqn. |
 | 00060  32 30 31 34 2d 30 38 2e  6f 72 67 2e 6e 76 6d 65  2014-08.org.nvme |
 | 00070  78 70 72 65 73 73 3a 4e  56 4d 66 3a 75 75 69 64  xpress:NVMf:uuid |
 | 00080  3a 64 34 35 32 39 37 64  34 2d 38 38 62 31 2d 34  :d45297d4-88b1-4 |
 | 00090  34 65 63 2d 61 37 36 33  2d 61 61 30 37 32 39 35  4ec-a763-aa07295 |
 | 000a0  36 34 32 37 33 0a 2c 71  75 65 75 65 5f 73 69 7a  64273.,queue_siz |
 | 000b0  65 3d 36 34                                       e=64             |
write(3, "1", 1)                        = 1
 | 00000  31                                                1                |
write(3, "nqn=nvmf-test,hostnqn=nqn.2014-0"..., 139) = 139
 | 00000  6e 71 6e 3d 6e 76 6d 66  2d 74 65 73 74 2c 68 6f  nqn=nvmf-test,ho |
 | 00010  73 74 6e 71 6e 3d 6e 71  6e 2e 32 30 31 34 2d 30  stnqn=nqn.2014-0 |
 | 00020  38 2e 6f 72 67 2e 6e 76  6d 65 78 70 72 65 73 73  8.org.nvmexpress |
 | 00030  3a 4e 56 4d 66 3a 75 75  69 64 3a 64 34 35 32 39  :NVMf:uuid:d4529 |
 | 00040  37 64 34 2d 38 38 62 31  2d 34 34 65 63 2d 61 37  7d4-88b1-44ec-a7 |
 | 00050  36 33 2d 61 61 30 37 32  39 35 36 34 32 37 33 0a  63-aa0729564273. |
 | 00060  2c 74 72 61 6e 73 70 6f  72 74 3d 72 64 6d 61 2c  ,transport=rdma, |
 | 00070  74 72 61 64 64 72 3d 31  2e 31 2e 31 2e 33 2c 74  traddr=1.1.1.3,t |
 | 00080  72 73 76 63 69 64 3d 34  34 32 30                 rsvcid=4420      |
+++ exited with 0 +++

So set the size parameter requested needed as well and while we're add it also
propagate the nr_io_queues down.

Signed-off-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 fabrics.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/fabrics.c b/fabrics.c
index 4871e1759ed0..5dec9f4fab17 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -633,6 +633,20 @@ static int connect_ctrl(struct nvmf_disc_rsp_page_entry *e)
 		p += len;
 	}
 
+	if (cfg.queue_size) {
+		len = sprintf(p, ",queue_size=%s", cfg.queue_size);
+		if (len < 0)
+			return -EINVAL;
+		p += len;
+	}
+
+	if (cfg.nr_io_queues) {
+		len = sprintf(p, ",nr_io_queues=%s", cfg.nr_io_queues);
+		if (len < 0)
+			return -EINVAL;
+		p += len;
+	}
+
 	switch (e->trtype) {
 	case NVMF_TRTYPE_LOOP: /* loop */
 		len = sprintf(p, ",transport=loop");
@@ -835,6 +849,7 @@ int discover(const char *desc, int argc, char **argv, bool connect)
 		{"hostnqn",     'q', "LIST", CFG_STRING, &cfg.hostnqn,     required_argument, "user-defined hostnqn (if default not used)" },
 		{"hostid",      'I', "LIST", CFG_STRING, &cfg.hostid,      required_argument, "user-defined hostid (if default not used)"},
 		{"queue-size",  'Q', "LIST", CFG_STRING, &cfg.queue_size,  required_argument, "number of io queue elements to use (default 128)" },
+		{"nr-io-queues",'i', "LIST", CFG_STRING, &cfg.nr_io_queues,required_argument, "number of io queues to use (default is core count)" },
 		{"raw",         'r', "LIST", CFG_STRING, &cfg.raw,         required_argument, "raw output file" },
 		{NULL},
 	};
-- 
2.13.6



From jthumshirn at suse.de  Tue Oct 17 02:20:37 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Tue, 17 Oct 2017 11:20:37 +0200
Subject: [PATCH v2 1/8] lib/scatterlist: Introduce sgl_alloc() and
 sgl_free()
In-Reply-To: <20171016224940.1332-2-bart.vanassche@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-2-bart.vanassche@wdc.com>
Message-ID: <20171017092037.bf6avqzejrcd5d5z@linux-x5ow.site>


Thanks Bart,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Tue Oct 17 02:44:44 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 17 Oct 2017 12:44:44 +0300
Subject: [PATCH nvme-cli] fabrics: also set queue_size and nr_io_queues on
 connect-all connects
In-Reply-To: <20171017084838.6827-1-jthumshirn@suse.de>
References: <20171017084838.6827-1-jthumshirn@suse.de>
Message-ID: <128ca19e-e5ce-a8bf-8cd5-ac45143c859e@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimber.me>


From sagi at grimberg.me  Tue Oct 17 03:01:34 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 17 Oct 2017 13:01:34 +0300
Subject: v4.14-rc5 NVMeOF regression?
In-Reply-To: <1508192588.2493.60.camel@wdc.com>
References: <1508192588.2493.60.camel@wdc.com>
Message-ID: <82de6177-27f5-463e-3446-0305fd9f6790@grimberg.me>


> Hello,

Hi Bart, thanks for reporting

> It has been a while since I ran any NVMeOF tests. But when I tried to test
> the v4.14-rc5 NVMeOF drivers the output shown below appeared. Is this a
> known issue? The following test triggered these call stacks:

Yea, we probably need to silence those with:
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..d7504f180d93 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1988,7 +1988,7 @@ static void nvme_rdma_remove_one(struct ib_device 
*ib_device, void *client_data)
         struct nvme_rdma_ctrl *ctrl;

         /* Delete all controllers using this device */
-       mutex_lock(&nvme_rdma_ctrl_mutex);
+       mutex_lock_nested(&nvme_rdma_ctrl_mutex, SINGLE_DEPTH_NESTING);
         list_for_each_entry(ctrl, &nvme_rdma_ctrl_list, list) {
                 if (ctrl->device->dev != ib_device)
                         continue;
--

I'll compile with CONFIG_DEBUG_LOCK_ALLOC and check this, but if you
happen to beat me to it, I can use your Tested-by tag.


From sagi at grimberg.me  Tue Oct 17 03:34:55 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 17 Oct 2017 13:34:55 +0300
Subject: [PATCH nvmetcli 1/2] nvmetcli: Expose ls to dump UI configuration
Message-ID: <1508236496-23921-1-git-send-email-sagi@grimberg.me>

for running: nvmetcli ls

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 nvmetcli | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/nvmetcli b/nvmetcli
index 5378365eb91f..951b6431e6f0 100755
--- a/nvmetcli
+++ b/nvmetcli
@@ -554,6 +554,7 @@ def usage():
     print("syntax: %s save [file_to_save_to]" % sys.argv[0])
     print("        %s restore [file_to_restore_from]" % sys.argv[0])
     print("        %s clear" % sys.argv[0])
+    print("        %s ls" % sys.argv[0])
     sys.exit(-1)
 
 
@@ -577,7 +578,14 @@ def clear(unused):
     nvme.Root().clear_existing()
 
 
-funcs = dict(save=save, restore=restore, clear=clear)
+def ls(unused):
+    shell = configshell.shell.ConfigShell('~/.nvmetcli')
+    UIRootNode(shell)
+    shell.run_cmdline("ls")
+    sys.exit(0)
+
+
+funcs = dict(save=save, restore=restore, clear=clear, ls=ls)
 
 
 def main():
-- 
2.7.4



From sagi at grimberg.me  Tue Oct 17 03:34:56 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 17 Oct 2017 13:34:56 +0300
Subject: [PATCH nvmetcli 2/2] nvmetcli: decorate namespace,
 subsystem and port UI
In-Reply-To: <1508236496-23921-1-git-send-email-sagi@grimberg.me>
References: <1508236496-23921-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508236496-23921-2-git-send-email-sagi@grimberg.me>

Display some info on the on the UI node.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 nvmetcli | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/nvmetcli b/nvmetcli
index 951b6431e6f0..ae86e89b61aa 100755
--- a/nvmetcli
+++ b/nvmetcli
@@ -24,7 +24,10 @@ import os
 import sys
 import configshell_fb as configshell
 import nvmet as nvme
+import uuid
 
+def ngiud_set(nguid):
+    return any(c.isdigit() and c != '0' for c in nguid)
 
 class UINode(configshell.node.ConfigNode):
     def __init__(self, name, parent=None, cfnode=None, shell=None):
@@ -160,6 +163,13 @@ class UISubsystemNode(UINode):
         UINamespacesNode(self)
         UIAllowedHostsNode(self)
 
+    def summary(self):
+        info = []
+        info.append("version=" + self.cfnode.get_attr("attr","version"))
+        info.append("allow_any=" + self.cfnode.get_attr("attr","allow_any_host"))
+        info.append("serial=" + self.cfnode.get_attr("attr","serial"))
+        return (", ".join(info), True)
+
 
 class UINamespacesNode(UINode):
     def __init__(self, parent):
@@ -247,6 +257,19 @@ class UINamespaceNode(UINode):
                 raise configshell.ExecutionError(
                     "The Namespace could not be disabled.")
 
+    def summary(self):
+        info = []
+        info.append("path=" + self.cfnode.get_attr("device", "path"))
+        ns_uuid = self.cfnode.get_attr("device", "uuid")
+        if uuid.UUID(ns_uuid).int != 0:
+            info.append("uuid=" + str(ns_uuid))
+        ns_nguid = self.cfnode.get_attr("device", "nguid")
+        if ngiud_set(ns_nguid):
+            info.append("nguid=" + ns_nguid)
+        info.append("enabled" if self.cfnode.get_enable() else "disabled")
+        ns_enabled = self.cfnode.get_enable()
+        return (", ".join(info), True if ns_enabled == 1 else ns_enabled)
+
 
 class UIAllowedHostsNode(UINode):
     def __init__(self, parent):
@@ -361,6 +384,13 @@ class UIPortNode(UINode):
             return "enabled"
         return "disabled"
 
+    def summary(self):
+        info = []
+        info.append("traddr=" + self.cfnode.get_attr("addr", "traddr"))
+        info.append("trsvcid=" + self.cfnode.get_attr("addr", "trsvcid"))
+        info.append("trtype=" + self.cfnode.get_attr("addr", "trtype"))
+        return (", ".join(info), True)
+
 
 class UIPortSubsystemsNode(UINode):
     def __init__(self, parent):
-- 
2.7.4



From james.smart at broadcom.com  Tue Oct 17 07:19:30 2017
From: james.smart at broadcom.com (James Smart)
Date: Tue, 17 Oct 2017 07:19:30 -0700
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
In-Reply-To: <20171017064306.GA16133@infradead.org>
References: <20171016151820.14766-1-jsmart2021@gmail.com>
 <20171017064306.GA16133@infradead.org>
Message-ID: <65999d52-2f65-2d9c-4cc4-db9d17fd9cab@broadcom.com>

On 10/16/2017 11:43 PM, Christoph Hellwig wrote:
>>   	if (status)
>>   		nvmet_set_status(req, status);
>>   
>> +	spin_lock_irqsave(&req->sq->sqhd_lock, flags);
>>   	if (req->sq->size)
>>   		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
>>   	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
>> +	spin_unlock_irqrestore(&req->sq->sqhd_lock, flags);
> What performance impact does this have?   I'm really reluctant to put
> an irq disabling spinlock into a hot path for a feature that
> theoretically is in the spec but ignored by every host.
>
> I'd much rather play games with cmpxchg or similar here.

It's not ignored by every host. Its ignored by the linux host. There are 
other hosts out there that follow the spec and pay attention to sqhd - 
and the linux target would like to be used with them.

I'm open to other alternatives, but in the past, with pci adapter 
interfaces with similar "ring" index updates, the increment followed by 
modulo has always proven to be an issue for such tricks.

-- james



From Bart.VanAssche at wdc.com  Tue Oct 17 07:28:29 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Tue, 17 Oct 2017 14:28:29 +0000
Subject: [PATCH v2 7/8] scsi/pmcraid: Remove an unused structure member
In-Reply-To: <5ea64c10-aa3e-f0ee-cfba-e93b3d8e1428@suse.de>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-8-bart.vanassche@wdc.com>
 <5ea64c10-aa3e-f0ee-cfba-e93b3d8e1428@suse.de>
Message-ID: <1508250508.3596.0.camel@wdc.com>

On Tue, 2017-10-17 at 08:21 +0200, Hannes Reinecke wrote:
> On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> > Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
> > Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> > Cc: linux-scsi at vger.kernel.org
> > Cc: Martin K. Petersen <martin.petersen at oracle.com>
> > Cc: Anil Ravindranath <anil_ravindranath at pmc-sierra.com>
> > ---
> >  drivers/scsi/pmcraid.h | 1 -
> >  1 file changed, 1 deletion(-)
> > 
> > diff --git a/drivers/scsi/pmcraid.h b/drivers/scsi/pmcraid.h
> > index 8bfac72a242b..44da91712115 100644
> > --- a/drivers/scsi/pmcraid.h
> > +++ b/drivers/scsi/pmcraid.h
> > @@ -542,7 +542,6 @@ struct pmcraid_sglist {
> >  	u32 order;
> >  	u32 num_sg;
> >  	u32 num_dma_sg;
> > -	u32 buffer_len;
> >  	struct scatterlist scatterlist[1];
> >  };
> >  
> > 
> 
> This actually is the same story that we've had with ipr (and, looking at
> the code, those two drivers look awfully similar ...).
> pmcraid_sglist looks as if it's a hardware-dependent structure, so just
> removing one entry from the middle of a structure might not be a good idea.
> But this is something for the pmcraid folks to clarify.

Hello Hannes,

Sorry but I don't see how a structure that contains a struct scatterlist
could be hardware-dependent?

Thanks,

Bart.

From Bart.VanAssche at wdc.com  Tue Oct 17 07:31:11 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Tue, 17 Oct 2017 14:31:11 +0000
Subject: [PATCH v2 5/8] target: Use sgl_alloc_order() and sgl_free()
In-Reply-To: <c5e05f84-2602-5c3f-521e-6f35601011cf@suse.de>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-6-bart.vanassche@wdc.com>
 <c5e05f84-2602-5c3f-521e-6f35601011cf@suse.de>
Message-ID: <1508250670.3596.2.camel@wdc.com>

On Tue, 2017-10-17 at 08:14 +0200, Hannes Reinecke wrote:
> On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> > [ ... ]
> >  void target_free_sgl(struct scatterlist *sgl, int nents)
> >  {
> > -	struct scatterlist *sg;
> > -	int count;
> > -
> > -	for_each_sg(sgl, sg, nents, count)
> > -		__free_page(sg_page(sg));
> > -
> > -	kfree(sgl);
> > +	sgl_free(sgl);
> >  }
> >  EXPORT_SYMBOL(target_free_sgl);
> >  
> > @@ -2423,42 +2417,10 @@ int
> >  target_alloc_sgl(struct scatterlist **sgl, unsigned int *nents, u32 length,
> >  		 bool zero_page, bool chainable)
> >  {
> > -	[ ... ]
> > +	*sgl = sgl_alloc_order(length, 0, chainable, gfp, nents);
> > +	return *sgl ? 0 : -ENOMEM;
> >  }
> >  EXPORT_SYMBOL(target_alloc_sgl);
> >  
> > The calling convention from target_alloc_sgl() is decidedly dodgy.
> 
> Can't we convert it into returning the sgl, and remove the first parameter?

Hello Hannes,

Another option is to remove the target_alloc_sgl() and target_free_sgl() functions
and to make LIO target drivers call sgl_alloc_order() and sgl_free_order() directly.
Do you perhaps have a preference for one of these two approaches?

Thanks,

Bart.

From keith.busch at intel.com  Tue Oct 17 07:39:01 2017
From: keith.busch at intel.com (Keith Busch)
Date: Tue, 17 Oct 2017 08:39:01 -0600
Subject: [PATCH] nvme-cli: add duplicate_connect parameter support
In-Reply-To: <20171011003536.5996-1-jsmart2021@gmail.com>
References: <20171011003536.5996-1-jsmart2021@gmail.com>
Message-ID: <20171017143900.GA32530@localhost.localdomain>

On Tue, Oct 10, 2017 at 05:35:36PM -0700, James Smart wrote:
> Add support for the duplicate_connect parameter for the fabrics
> device.
> 
> Rather than add yet another 6-line code block, created a small helper
> routine invoked for each argument option.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>

Thanks, applied.


From keith.busch at intel.com  Tue Oct 17 07:39:12 2017
From: keith.busch at intel.com (Keith Busch)
Date: Tue, 17 Oct 2017 08:39:12 -0600
Subject: [PATCH nvme-cli] fabrics: also set queue_size and nr_io_queues
 on connect-all connects
In-Reply-To: <128ca19e-e5ce-a8bf-8cd5-ac45143c859e@grimberg.me>
References: <20171017084838.6827-1-jthumshirn@suse.de>
 <128ca19e-e5ce-a8bf-8cd5-ac45143c859e@grimberg.me>
Message-ID: <20171017143912.GB32530@localhost.localdomain>

Applied, thanks!


From Jeff.Lien at wdc.com  Tue Oct 17 08:34:19 2017
From: Jeff.Lien at wdc.com (Jeffrey Lien)
Date: Tue, 17 Oct 2017 15:34:19 +0000
Subject: Test Email
Message-ID: <MWHPR04MB1025BBFAF0881AF34A5A8A99EA4C0@MWHPR04MB1025.namprd04.prod.outlook.com>

Having troubles sending to this id.  This is a test to see if it makes it.  

Jeff Lien
Linux Device Driver Development
Device Host Apps and Drivers
jeff.lien at wdc.com
o: 507-322-2416 (ext. 23-2416)
m: 507-273-9124








From jeff.lien at hgst.com  Tue Oct 17 09:56:43 2017
From: jeff.lien at hgst.com (jeffreyalien)
Date: Tue, 17 Oct 2017 11:56:43 -0500
Subject: [PATCH] nvme-cli : wdc-plugin Add support for WDC SN100 and SN200
 devices.
Message-ID: <20171017165643.31988-1-jeff.lien@hgst.com>

From: jeffreyalien <jeff.lien at wdc.com>

Update the checking for the wdc-plugin commands so that a
"Device not supported" message will be displayed for non-WDC
devices.

Signed-off-by: jeffreyalien <jeff.lien at wdc.com>
---
 wdc-nvme.c |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/wdc-nvme.c b/wdc-nvme.c
index 03c86c9..b981298 100644
--- a/wdc-nvme.c
+++ b/wdc-nvme.c
@@ -47,9 +47,11 @@
 #define WDC_NVME_SUBCMD_SHIFT	8
 
 #define WDC_NVME_LOG_SIZE_DATA_LEN			0x08
+
 /* Device Config */
-#define WDC_NVME_GF_VID		0x1c58
-#define WDC_NVME_GF_CNTL_ID 0x0003
+#define WDC_NVME_VID  			0x1c58
+#define WDC_NVME_SN100_CNTL_ID	0x0003
+#define WDC_NVME_SN200_CNTL_ID	0x0023
 
 /* Capture Diagnostics */
 #define WDC_NVME_CAP_DIAG_HEADER_TOC_SIZE	WDC_NVME_LOG_SIZE_DATA_LEN
@@ -200,9 +202,10 @@ static int wdc_check_device(int fd)
 		return -1;
 	}
 	ret = -1;
-	/* GF : ctrl->cntlid == PCI Device ID, use that with VID to identify GF Device */
-	if ((le32_to_cpu(ctrl.cntlid) == WDC_NVME_GF_CNTL_ID) &&
-			(le32_to_cpu(ctrl.vid) == WDC_NVME_GF_VID))
+	/* WDC : ctrl->cntlid == PCI Device ID, use that with VID to identify WDC Devices */
+	if ((le32_to_cpu(ctrl.vid) == WDC_NVME_VID) &&
+		((le32_to_cpu(ctrl.cntlid) == WDC_NVME_SN100_CNTL_ID) ||
+		(le32_to_cpu(ctrl.cntlid) == WDC_NVME_SN200_CNTL_ID)))
 		ret = 0;
 	else
 		fprintf(stderr, "WARNING : WDC : Device not supported\n");
-- 
1.7.1



From pmenzel at molgen.mpg.de  Tue Oct 17 13:33:56 2017
From: pmenzel at molgen.mpg.de (Paul Menzel)
Date: Tue, 17 Oct 2017 22:33:56 +0200
Subject: How to improve long suspend time in `nvme_shutdown_ctrl`
Message-ID: <72b666c9-dd57-ef87-e89e-ba280f34e90c@molgen.mpg.de>

Dear Linux folks,


Instrumenting ACPI S3 suspend time with AnalyzeSusend [1] on Linux 
4.14-rc5 on a TUXEDO Book BU1406 shows that 338 ms are spend in nvme.

> nvme @ 0000:04:00.0 {nvme} async_device (Total Suspend: 356.823 ms Total Resume: 20.148 ms)
```
$ lspci -nn -s 4:00.0
04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co 
Ltd NVMe SSD Controller SM961/PM961 [144d:a804]$ sudo 
./analyze_suspend.py -config config/suspend-callgraph.cfg
```

 From the HTML output:

> nvme_shutdown_ctrl [nvme_core] (321.044 ms @ 476.668326)
> ? nvme_pci_reg_write32 [nvme] (0.000 ms @ 476.668327)
>   nvme_pci_reg_read32 [nvme] (0.033 ms @ 476.668329)
>   msleep (104.991 ms @ 476.668363)
>   nvme_pci_reg_read32 [nvme] (0.004 ms @ 476.773356)
>   msleep (107.992 ms @ 476.773361)
>   nvme_pci_reg_read32 [nvme] (0.001 ms @ 476.881355)
>   msleep (108.009 ms @ 476.881357)
>   nvme_pci_reg_read32 [nvme] (0.001 ms @ 476.989367)

Is that the right register to read? Can this be improved? If not, is 
that a firmware bug?


Kind regards,

Paul


[1] https://github.com/01org/pm-graph


From jsmart2021 at gmail.com  Tue Oct 17 16:32:43 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:43 -0700
Subject: [PATCH v3 0/5] nvme_fc: add dev_loss_tmo support
Message-ID: <20171017233248.6769-1-jsmart2021@gmail.com>

FC, on the SCSI side, has long had a device loss timeout which governed
how long it would hide connectivity loss to remote target ports.
The timeout value is maintained in the SCSI FC transport and admins
are used to going there to maintain it.

Eventually, the SCSI FC transport will be moved into something
independent from and above SCSI so that SCSI and NVME protocols can
be peers. In the meantime, to add the functionality now, and sync with
the SCSI FC transport, the LLDD will be used as the conduit. The
initial value for the timeout can be set by the LLDD when it creates
the remoteport via nvme_fc_register_remoteport(). Later, if the value
is updated via the SCSI transport, the LLDD can call a new nvme_fc
routine to update the remoteport's dev_loss_tmo value.

The nvme fabrics implementation already has a similar timer, the
ctrl_loss_tmo, which is distilled into a max_reconnect attempts and
a reconnect_delay between attempts, where the overall duration until
max is hit is the ctrl_loss_tmo.  This was primarily for transports
that didn't have the ability to track device connectivity and would
retry per the delay until finally giving up.

The implementation in this patch set implements the FC dev_loss_tmo
at the FC port level. The timer is initiated when connectivity is
lost. If connectivity is not re-established and the timer expires,
all controllers on the remote port will be deleted.

When the FC remoteport-level connectivity is lost, all controllers
on the remoteport are reset, which results in them transitioning to
a reconnecting state, with the ctrl_loss_tmo behavior kicks in. Thus
the controller may be deleted as soon as ctrl_loss_tmo expires or
the FC port level dev_loss_tmo expires.

If connectivity is re-established before the dev_loss_tmo expires,
any controllers on the remoteport, which would be in a reconnecting
state, would immediately have a reconnect attempted.

The patches were cut on the nvme-4.15 branch
Patch 5, which adds the dev_loss_tmo timeout, is dependent on the
nvme_fc_signal_discovery_scan() routine added by this patch:
http://lists.infradead.org/pipermail/linux-nvme/2017-September/012781.html
The patch has been approved but not yet pulled into a tree.

V3:
 In v2, the implementation merged the dev_loss_tmo value into the
 ctlr_loss_tmo in the controller, so only a single timer on each controller
 was running.
 V3 changed to keep the dev_loss_tmo on the FC remoteport and to run it
 independently from the ctrl_loss_tmo timer, excepting for loss of
 connectivity to start both simultaneously.


James Smart (5):
  nvme core: allow controller RESETTING to RECONNECTING transition
  nvme_fc: change ctlr state assignments during reset/reconnect
  nvme_fc: add a dev_loss_tmo field to the remoteport
  nvme_fc: check connectivity before initiating reconnects
  nvme_fc: add dev_loss_tmo timeout and remoteport resume support

 drivers/nvme/host/core.c       |   1 +
 drivers/nvme/host/fc.c         | 337 ++++++++++++++++++++++++++++++++++++-----
 include/linux/nvme-fc-driver.h |  11 +-
 3 files changed, 310 insertions(+), 39 deletions(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 17 16:32:44 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:44 -0700
Subject: [PATCH v3 1/5] nvme core: allow controller RESETTING to RECONNECTING
 transition
In-Reply-To: <20171017233248.6769-1-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
Message-ID: <20171017233248.6769-2-jsmart2021@gmail.com>

Allow controller state transition : RESETTING to RECONNECTING

Transport will typically transition from LIVE->RESETTING when
initially performing a reset or recovering from an error. Adding
this transition allows a transport to transition to RECONNECTING
when it checks/waits for connectivity then creates new transport
connections and reinits the controller.

-- james

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>

---
v3: no change. incorporated Reviewed-by's
---
 drivers/nvme/host/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 26c8913435b2..aa9e2df27bf7 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -205,6 +205,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 	case NVME_CTRL_RECONNECTING:
 		switch (old_state) {
 		case NVME_CTRL_LIVE:
+		case NVME_CTRL_RESETTING:
 			changed = true;
 			/* FALLTHRU */
 		default:
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 17 16:32:46 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:46 -0700
Subject: [PATCH v3 3/5] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <20171017233248.6769-1-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
Message-ID: <20171017233248.6769-4-jsmart2021@gmail.com>

Add a dev_loss_tmo value, paralleling the SCSI FC transport, for device
connectivity loss.

The transport initializes the value in the nvme_fc_register_remoteport()
call. If the value is not set, a default of 60s is set.

Add a new routine to the api, nvme_fc_set_remoteport_devloss() routine,
which allows the lldd to dynamically update the value on an existing
remoteport.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v3:
 Removed the expected reconnect time and min devloss defines. They
 were not that meaningful.
 Incorporated the new nvme_fc_set_remoteport_devloss() routine
 which was in a different patch in v2. New routine no longer
 touches controllers. It only modifies the remoteport.
---
 drivers/nvme/host/fc.c         | 31 +++++++++++++++++++++++++++++++
 include/linux/nvme-fc-driver.h | 11 +++++++++--
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 5afb518c39ad..3ac49e670c38 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -45,6 +45,8 @@ enum nvme_fc_queue_flags {
 
 #define NVMEFC_QUEUE_DELAY	3		/* ms units */
 
+#define NVME_FC_DEFAULT_DEV_LOSS_TMO	60	/* seconds */
+
 struct nvme_fc_queue {
 	struct nvme_fc_ctrl	*ctrl;
 	struct device		*dev;
@@ -587,6 +589,11 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	newrec->remoteport.port_id = pinfo->port_id;
 	newrec->remoteport.port_state = FC_OBJSTATE_ONLINE;
 	newrec->remoteport.port_num = idx;
+	/* a registration value of dev_loss_tmo=0 results in the default */
+	if (pinfo->dev_loss_tmo)
+		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
+	else
+		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
 
 	spin_lock_irqsave(&nvme_fc_lock, flags);
 	list_add_tail(&newrec->endp_list, &lport->endp_list);
@@ -690,6 +697,30 @@ nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport)
 }
 EXPORT_SYMBOL_GPL(nvme_fc_rescan_remoteport);
 
+int
+nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
+			u32 dev_loss_tmo)
+{
+	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rport->lock, flags);
+
+	if (portptr->port_state != FC_OBJSTATE_ONLINE) {
+		spin_unlock_irqrestore(&rport->lock, flags);
+		return -EINVAL;
+	}
+
+	/* a dev_loss_tmo of 0 (immediate) is allowed to be set */
+	rport->remoteport.dev_loss_tmo = dev_loss_tmo;
+
+	spin_unlock_irqrestore(&rport->lock, flags);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_fc_set_remoteport_devloss);
+
 
 /* *********************** FC-NVME DMA Handling **************************** */
 
diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
index 4ea03b9a5c8c..3de268bf86bf 100644
--- a/include/linux/nvme-fc-driver.h
+++ b/include/linux/nvme-fc-driver.h
@@ -40,6 +40,8 @@
  * @node_name: FC WWNN for the port
  * @port_name: FC WWPN for the port
  * @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx)
+ * @dev_loss_tmo: maximum delay for reconnects to an association on
+ *             this device. Used only on a remoteport.
  *
  * Initialization values for dynamic port fields:
  * @port_id:      FC N_Port_ID currently assigned the port. Upper 8 bits must
@@ -50,6 +52,7 @@ struct nvme_fc_port_info {
 	u64			port_name;
 	u32			port_role;
 	u32			port_id;
+	u32			dev_loss_tmo;
 };
 
 
@@ -202,6 +205,9 @@ enum nvme_fc_obj_state {
  *             The length of the buffer corresponds to the local_priv_sz
  *             value specified in the nvme_fc_port_template supplied by
  *             the LLDD.
+ * @dev_loss_tmo: maximum delay for reconnects to an association on
+ *             this device. To modify, lldd must call
+ *             nvme_fc_set_remoteport_devloss().
  *
  * Fields with dynamic values. Values may change base on link state. LLDD
  * may reference fields directly to change them. Initialized by the
@@ -259,10 +265,9 @@ struct nvme_fc_remote_port {
 	u32 port_role;
 	u64 node_name;
 	u64 port_name;
-
 	struct nvme_fc_local_port *localport;
-
 	void *private;
+	u32 dev_loss_tmo;
 
 	/* dynamic fields */
 	u32 port_id;
@@ -448,6 +453,8 @@ int nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *remoteport);
 
 void nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport);
 
+int nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *remoteport,
+			u32 dev_loss_tmo);
 
 
 /*
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 17 16:32:45 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:45 -0700
Subject: [PATCH v3 2/5] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20171017233248.6769-1-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
Message-ID: <20171017233248.6769-3-jsmart2021@gmail.com>

Clean up some of the controller state checks and add the
RESETTING->RECONNECTING state transition.

Specifically:
- the movement of the RESETTING state change and schedule of
  reset_work to core doesn't work wiht nvme_fc_error_recovery
  setting state to RECONNECTING before attempting to reset.
  Remove the state change as the reset request does it.
- In the rare cases where an error occurs right as we're
  transitioning to LIVE, defer the controller start actions.
- In error handling on teardown of associations while
  performing initial controller creation - avoid quiesce
  calls on the admin_q. They are unneeded.
- Add the RESETTING->RECONNECTING transition in the reset
  handler.

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Christoph Hellwig <hch at lst.de>

---
v3: no change. incorporated Reviewed-by's
---
 drivers/nvme/host/fc.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 48c241620d2b..5afb518c39ad 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -1881,13 +1881,6 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 	dev_warn(ctrl->ctrl.device,
 		"NVME-FC{%d}: resetting controller\n", ctrl->cnum);
 
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
-		dev_err(ctrl->ctrl.device,
-			"NVME-FC{%d}: error_recovery: Couldn't change state "
-			"to RECONNECTING\n", ctrl->cnum);
-		return;
-	}
-
 	nvme_reset_ctrl(&ctrl->ctrl);
 }
 
@@ -2521,11 +2514,11 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
-	WARN_ON_ONCE(!changed);
 
 	ctrl->ctrl.nr_reconnects = 0;
 
-	nvme_start_ctrl(&ctrl->ctrl);
+	if (changed)
+		nvme_start_ctrl(&ctrl->ctrl);
 
 	return 0;	/* Success */
 
@@ -2593,7 +2586,8 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 	 * use blk_mq_tagset_busy_itr() and the transport routine to
 	 * terminate the exchanges.
 	 */
-	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	if (ctrl->ctrl.state != NVME_CTRL_NEW)
+		blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 
@@ -2697,12 +2691,8 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 static void
 nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 {
-	/* If we are resetting/deleting then do nothing */
-	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING) {
-		WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW ||
-			ctrl->ctrl.state == NVME_CTRL_LIVE);
+	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING)
 		return;
-	}
 
 	dev_info(ctrl->ctrl.device,
 		"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
@@ -2731,9 +2721,17 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	int ret;
 
 	nvme_stop_ctrl(&ctrl->ctrl);
+
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
+		dev_err(ctrl->ctrl.device,
+			"NVME-FC{%d}: error_recovery: Couldn't change state "
+			"to RECONNECTING\n", ctrl->cnum);
+		return;
+	}
+
 	ret = nvme_fc_create_association(ctrl);
 	if (ret)
 		nvme_fc_reconnect_or_delete(ctrl, ret);
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 17 16:32:47 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:47 -0700
Subject: [PATCH v3 4/5] nvme_fc: check connectivity before initiating
 reconnects
In-Reply-To: <20171017233248.6769-1-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
Message-ID: <20171017233248.6769-5-jsmart2021@gmail.com>

check remoteport connectivity before initiating reconnects

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v3:
  removed inline with lock around 1 line read of state.
  replaced calls with explicit checks of read state
---
 drivers/nvme/host/fc.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 3ac49e670c38..61da2f92f71a 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -810,7 +810,6 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
 		dma_unmap_sg(dev, sg, nents, dir);
 }
 
-
 /* *********************** FC-NVME LS Handling **************************** */
 
 static void nvme_fc_ctrl_put(struct nvme_fc_ctrl *);
@@ -2454,6 +2453,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 
 	++ctrl->ctrl.nr_reconnects;
 
+	if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+		return -ENODEV;
+
 	/*
 	 * Create the admin queue
 	 */
@@ -2730,6 +2732,10 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 		ctrl->cnum, status);
 
 	if (nvmf_should_reconnect(&ctrl->ctrl)) {
+		/* Only schedule the reconnect if the remote port is online */
+		if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+			return;
+
 		dev_info(ctrl->ctrl.device,
 			"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
 			ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
@@ -2763,12 +2769,15 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	ret = nvme_fc_create_association(ctrl);
-	if (ret)
-		nvme_fc_reconnect_or_delete(ctrl, ret);
-	else
-		dev_info(ctrl->ctrl.device,
-			"NVME-FC{%d}: controller reset complete\n", ctrl->cnum);
+	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) {
+		ret = nvme_fc_create_association(ctrl);
+		if (ret)
+			nvme_fc_reconnect_or_delete(ctrl, ret);
+		else
+			dev_info(ctrl->ctrl.device,
+				"NVME-FC{%d}: controller reset complete\n",
+				ctrl->cnum);
+	}
 }
 
 static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
-- 
2.13.1



From jsmart2021 at gmail.com  Tue Oct 17 16:32:48 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 17 Oct 2017 16:32:48 -0700
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20171017233248.6769-1-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
Message-ID: <20171017233248.6769-6-jsmart2021@gmail.com>

This patch adds the dev_loss_tmo functionality to the transport.

When a remoteport is unregistered (connectivity lost), the following
actions are taken:
- the remoteport is marked DELETED
- a dev_loss_tmo timer is started for the remoteport
- all controllers on the remoteport are reset.

After a controller resets, it will stall in a RECONNECTING state
waiting for one of the following:
- the controller will continue to attempt reconnect per max_retries
  and reconnect_delay. As no remoteport connectivity, the reconnect
  attempt will immediately fail. if max reconnect attempts are
  reached (e.g. ctrl_loss_tmo reached) the controller is deleted.
- the remoteport is re-registered prior to dev_loss_tmo expiring.
  The resume of the remoteport will immediately attempt to reconnect
  each of its suspended controllers.
- the remoteport's dev_loss_tmo expires, causing all of its
  controllers to be deleted.

Signed-off-by: James Smart <james.smart at broadcom.com>

v3:
  Reworked so dev_loss_tmo specific to the remoteport. Revised
  so connectivity loss resets controllers, and connectivity gain
  schedules reconnect.
---
 drivers/nvme/host/fc.c | 289 +++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 257 insertions(+), 32 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 61da2f92f71a..3628790371d7 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -138,6 +138,7 @@ struct nvme_fc_rport {
 	struct nvme_fc_lport		*lport;
 	spinlock_t			lock;
 	struct kref			ref;
+	struct delayed_work		dev_loss_work;
 } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
 
 enum nvme_fcctrl_flags {
@@ -503,6 +504,8 @@ nvme_fc_free_rport(struct kref *ref)
 	WARN_ON(rport->remoteport.port_state != FC_OBJSTATE_DELETED);
 	WARN_ON(!list_empty(&rport->ctrl_list));
 
+	cancel_delayed_work_sync(&rport->dev_loss_work);
+
 	/* remove from lport list */
 	spin_lock_irqsave(&nvme_fc_lock, flags);
 	list_del(&rport->endp_list);
@@ -530,6 +533,124 @@ nvme_fc_rport_get(struct nvme_fc_rport *rport)
 	return kref_get_unless_zero(&rport->ref);
 }
 
+static void
+nvme_fc_resume_controller(struct nvme_fc_ctrl *ctrl)
+{
+	switch (ctrl->ctrl.state) {
+	case NVME_CTRL_NEW:
+	case NVME_CTRL_RECONNECTING:
+		/*
+		 * As all reconnects were suppressed, schedule a
+		 * connect.
+		 */
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: connectivity re-established. "
+			"Attempting reconnect\n", ctrl->cnum);
+
+		queue_delayed_work(nvme_wq, &ctrl->connect_work, 0);
+		break;
+
+	case NVME_CTRL_RESETTING:
+		/*
+		 * Controller is already in the process of terminating the
+		 * association. No need to do anything further. The reconnect
+		 * step will naturally occur after the reset completes.
+		 */
+		break;
+
+	default:
+		/* no action to take - let it delete */
+		break;
+	}
+}
+
+static struct nvme_fc_rport *
+nvme_fc_attach_to_suspended_rport(struct nvme_fc_lport *lport,
+				struct nvme_fc_port_info *pinfo)
+{
+	struct nvme_fc_rport *rport;
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+
+	spin_lock_irqsave(&nvme_fc_lock, flags);
+
+	list_for_each_entry(rport, &lport->endp_list, endp_list) {
+		if (rport->remoteport.node_name != pinfo->node_name ||
+		    rport->remoteport.port_name != pinfo->port_name)
+			continue;
+
+		if (!nvme_fc_rport_get(rport)) {
+			rport = ERR_PTR(-ENOLCK);
+			goto out_done;
+		}
+
+		spin_unlock_irqrestore(&nvme_fc_lock, flags);
+
+		cancel_delayed_work_sync(&rport->dev_loss_work);
+
+		spin_lock_irqsave(&rport->lock, flags);
+
+		/* has it been unregistered */
+		if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
+			/* means lldd called us twice */
+			spin_unlock_irqrestore(&rport->lock, flags);
+			nvme_fc_rport_put(rport);
+			return ERR_PTR(-ESTALE);
+		}
+
+		rport->remoteport.port_state = FC_OBJSTATE_ONLINE;
+
+		/*
+		 * kick off a reconnect attempt on all associations to the
+		 * remote port. A successful reconnects will resume i/o.
+		 */
+		list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
+			nvme_fc_resume_controller(ctrl);
+
+		spin_unlock_irqrestore(&rport->lock, flags);
+
+		return rport;
+	}
+
+	rport = NULL;
+
+out_done:
+	spin_unlock_irqrestore(&nvme_fc_lock, flags);
+
+	return rport;
+}
+
+static void
+nvme_fc_rport_dev_loss_work(struct work_struct *work)
+{
+	struct nvme_fc_rport *rport =
+			container_of(to_delayed_work(work),
+				struct nvme_fc_rport, dev_loss_work);
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rport->lock, flags);
+
+	/* If port state transitioned dev loss shouldn't kick in */
+	if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
+		spin_unlock_irqrestore(&rport->lock, flags);
+		return;
+	}
+
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
+		dev_warn(ctrl->ctrl.device,
+			"NVME-FC{%d}: Remote Port failed to reconnect within "
+			"dev_loss_tmo (%d seconds). Deleting controller\n",
+			ctrl->cnum, rport->remoteport.dev_loss_tmo);
+		if (__nvme_fc_del_ctrl(ctrl))
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: delete request failed\n",
+				ctrl->cnum);
+	}
+
+	spin_unlock_irqrestore(&rport->lock, flags);
+}
+
 /**
  * nvme_fc_register_remoteport - transport entry point called by an
  *                              LLDD to register the existence of a NVME
@@ -556,22 +677,46 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	unsigned long flags;
 	int ret, idx;
 
+	if (!nvme_fc_lport_get(lport)) {
+		ret = -ESHUTDOWN;
+		goto out_reghost_failed;
+	}
+
+	/*
+	 * look to see if there is already a remoteport that is waiting
+	 * for a reconnect (within dev_loss_tmo) with the same WWN's.
+	 * If so, transition to it and reconnect.
+	 */
+	newrec = nvme_fc_attach_to_suspended_rport(lport, pinfo);
+
+	/* found an rport, but something about its state is bad */
+	if (IS_ERR(newrec)) {
+		ret = PTR_ERR(newrec);
+		goto out_lport_put;
+
+	/* found existing rport, which was resumed */
+	} else if (newrec) {
+		/* Ignore pinfo->dev_loss_tmo. Leave rport and ctlr's as is */
+
+		nvme_fc_lport_put(lport);
+		nvme_fc_signal_discovery_scan(lport, newrec);
+		*portptr = &newrec->remoteport;
+		return 0;
+	}
+
+	/* nothing found - allocate a new remoteport struct */
+
 	newrec = kmalloc((sizeof(*newrec) + lport->ops->remote_priv_sz),
 			 GFP_KERNEL);
 	if (!newrec) {
 		ret = -ENOMEM;
-		goto out_reghost_failed;
-	}
-
-	if (!nvme_fc_lport_get(lport)) {
-		ret = -ESHUTDOWN;
-		goto out_kfree_rport;
+		goto out_lport_put;
 	}
 
 	idx = ida_simple_get(&lport->endp_cnt, 0, 0, GFP_KERNEL);
 	if (idx < 0) {
 		ret = -ENOSPC;
-		goto out_lport_put;
+		goto out_kfree_rport;
 	}
 
 	INIT_LIST_HEAD(&newrec->endp_list);
@@ -594,6 +739,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
 	else
 		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
+	INIT_DELAYED_WORK(&newrec->dev_loss_work, nvme_fc_rport_dev_loss_work);
 
 	spin_lock_irqsave(&nvme_fc_lock, flags);
 	list_add_tail(&newrec->endp_list, &lport->endp_list);
@@ -604,10 +750,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	*portptr = &newrec->remoteport;
 	return 0;
 
-out_lport_put:
-	nvme_fc_lport_put(lport);
 out_kfree_rport:
 	kfree(newrec);
+out_lport_put:
+	nvme_fc_lport_put(lport);
 out_reghost_failed:
 	*portptr = NULL;
 	return ret;
@@ -638,6 +784,61 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
 	return 0;
 }
 
+static void
+nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
+{
+	dev_info(ctrl->ctrl.device,
+		"NVME-FC{%d}: controller connectivity lost. Awaiting "
+		"Reconnect", ctrl->cnum);
+
+	switch (ctrl->ctrl.state) {
+	case NVME_CTRL_NEW:
+	case NVME_CTRL_LIVE:
+		/*
+		 * Schedule a controller reset. The reset will
+		 * terminate the association and schedule the
+		 * reconnect timer. Reconnects will be attempted
+		 * until either the ctlr_loss_tmo
+		 * (max_retries * connect_delay) expires or the
+		 * remoteport's dev_loss_tmo expires.
+		 */
+		if (nvme_reset_ctrl(&ctrl->ctrl)) {
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: Couldn't schedule reset. "
+				"Deleting controller.\n",
+				ctrl->cnum);
+			__nvme_fc_del_ctrl(ctrl);
+		}
+		break;
+
+	case NVME_CTRL_RECONNECTING:
+		/*
+		 * The association has already been terminated
+		 * and the controller is attempting reconnects.
+		 * No need to do anything futher. Reconnects will
+		 * be attempted until either the ctlr_loss_tmo
+		 * (max_retries * connect_delay) expires or the
+		 * remoteport's dev_loss_tmo expires.
+		 */
+		break;
+
+	case NVME_CTRL_RESETTING:
+		/*
+		 * Controller is already in the process of
+		 * terminating the association. No need to do
+		 * anything further. The reconnect step will
+		 * kick in naturally after the association is
+		 * terminated.
+		 */
+		break;
+
+	case NVME_CTRL_DELETING:
+	default:
+		/* no action to take - let it delete */
+		break;
+	}
+}
+
 /**
  * nvme_fc_unregister_remoteport - transport entry point called by an
  *                              LLDD to deregister/remove a previously
@@ -667,15 +868,32 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
 	}
 	portptr->port_state = FC_OBJSTATE_DELETED;
 
-	/* tear down all associations to the remote port */
-	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
-		__nvme_fc_del_ctrl(ctrl);
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
+		/* if dev_loss_tmo==0, dev loss is immediate */
+		if (!portptr->dev_loss_tmo) {
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: controller connectivity lost. "
+				"Deleting controller.\n",
+				ctrl->cnum);
+			__nvme_fc_del_ctrl(ctrl);
+		} else
+			nvme_fc_ctrl_connectivity_loss(ctrl);
+	}
 
 	spin_unlock_irqrestore(&rport->lock, flags);
 
 	nvme_fc_abort_lsops(rport);
 
+	queue_delayed_work(nvme_wq, &rport->dev_loss_work,
+			portptr->dev_loss_tmo * HZ);
+
+	/*
+	 * release the reference, which will allow, if all controllers
+	 * go away, which should only occur after dev_loss_tmo occurs,
+	 * for the rport to be torn down.
+	 */
 	nvme_fc_rport_put(rport);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_fc_unregister_remoteport);
@@ -702,7 +920,6 @@ nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
 			u32 dev_loss_tmo)
 {
 	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
-	struct nvme_fc_ctrl *ctrl;
 	unsigned long flags;
 
 	spin_lock_irqsave(&rport->lock, flags);
@@ -2727,25 +2944,31 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING)
 		return;
 
-	dev_info(ctrl->ctrl.device,
-		"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
-		ctrl->cnum, status);
+	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
+			ctrl->cnum, status);
 
 	if (nvmf_should_reconnect(&ctrl->ctrl)) {
-		/* Only schedule the reconnect if the remote port is online */
-		if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
-			return;
-
-		dev_info(ctrl->ctrl.device,
-			"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
-			ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
+		if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
+			dev_info(ctrl->ctrl.device,
+				"NVME-FC{%d}: Reconnect attempt in %d "
+				"seconds.\n",
+				ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
 		queue_delayed_work(nvme_wq, &ctrl->connect_work,
 				ctrl->ctrl.opts->reconnect_delay * HZ);
 	} else {
-		dev_warn(ctrl->ctrl.device,
+		if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
+			dev_warn(ctrl->ctrl.device,
 				"NVME-FC{%d}: Max reconnect attempts (%d) "
 				"reached. Removing controller\n",
 				ctrl->cnum, ctrl->ctrl.nr_reconnects);
+		else
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: Max reconnect attempts (%d) "
+				"reached while waiting for remoteport "
+				"connectivity. Removing controller\n",
+				ctrl->cnum, ctrl->ctrl.nr_reconnects);
 		WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
 	}
 }
@@ -2769,15 +2992,17 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) {
+	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
 		ret = nvme_fc_create_association(ctrl);
-		if (ret)
-			nvme_fc_reconnect_or_delete(ctrl, ret);
-		else
-			dev_info(ctrl->ctrl.device,
-				"NVME-FC{%d}: controller reset complete\n",
-				ctrl->cnum);
-	}
+	else
+		ret = -ENOTCONN;
+
+	if (ret)
+		nvme_fc_reconnect_or_delete(ctrl, ret);
+	else
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: controller reset complete\n",
+			ctrl->cnum);
 }
 
 static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
-- 
2.13.1



From drake at endlessm.com  Tue Oct 17 20:01:33 2017
From: drake at endlessm.com (Daniel Drake)
Date: Wed, 18 Oct 2017 11:01:33 +0800
Subject: AHCI NVMe remap update
Message-ID: <CAD8Lp450TSnpzTAGiAgKjJws6VnVu9ZWv_x-GFQ5RhNOTnNCpQ@mail.gmail.com>

Hi Dan,

Thanks for the patches you posted last year at:
  http://lists.infradead.org/pipermail/linux-nvme/2016-October/006800.html
  [PATCH 0/5] ahci: nvme remap support

Looks like these patches stalled because of a lack of clarity of how
the device operates. Has there been any movement since then?

The issue became less critical soon after when Lenovo published a BIOS
update for affected models allowing the user to go into the BIOS and
disable RAID mode - it looked like it might have been a one off
headache for some models.

However it appears that Lenovo's newer models are still coming with
the RAID mode by default (we have seen this on Yoga 720), and asking
ordinary users to go into the BIOS, disable RAID and lose access to
their Windows install is not always a feasible option (we're trying to
support dual boot here).

Would be great if you could take another look at this.

Thanks
Daniel


From sagi at grimberg.me  Tue Oct 17 22:26:05 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 08:26:05 +0300
Subject: v4.14-rc5 NVMeOF regression?
In-Reply-To: <5fb38923-36f7-c069-5f1d-96f4a9c98248@wdc.com>
References: <1508192588.2493.60.camel@wdc.com>
 <82de6177-27f5-463e-3446-0305fd9f6790@grimberg.me>
 <5fb38923-36f7-c069-5f1d-96f4a9c98248@wdc.com>
Message-ID: <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>


> Hello Sagi,
> 
> Had I mentioned that I ran into a real deadlock and not only into a 
> lockdep complaint?

No, and the output is a lockdep complaint which I thought needed
silencing.

If you ran into a real deadlock, did you have any other output from
hung_task watchdog? I do not yet understand the root cause from
lockdep info provided.

Also, do you know at which test-case this happened?

> Additionally, can you explain to me why you think the 
> above patch is necessary? As far as I can see nvme_rdma_ctrl_mutex is 
> not held around the ib_register_client(&nvme_rdma_ib_client) call nor 
> around the ib_unregister_client(&nvme_rdma_ib_client) calls.

Its held under ib_unregister_client, my understanding is that the
_nested annotation is required for the inner mutex lock acquire.


From hch at infradead.org  Tue Oct 17 23:30:27 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 17 Oct 2017 23:30:27 -0700
Subject: AHCI NVMe remap update
In-Reply-To: <CAD8Lp450TSnpzTAGiAgKjJws6VnVu9ZWv_x-GFQ5RhNOTnNCpQ@mail.gmail.com>
References: <CAD8Lp450TSnpzTAGiAgKjJws6VnVu9ZWv_x-GFQ5RhNOTnNCpQ@mail.gmail.com>
Message-ID: <20171018063027.GA12755@infradead.org>

On Wed, Oct 18, 2017 at 11:01:33AM +0800, Daniel Drake wrote:
> Would be great if you could take another look at this.

The right way to do this would be to expose a fake PCIe root port
that both the AHCI and NVMe driver bind to.  But only Intel has the
detailed information to do this.  Especially as we'd have to do
things like recovering the exact PCI ID for quirks in the NVMe driver
to make this whole scheme work reliably.


From hch at infradead.org  Tue Oct 17 23:55:33 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 17 Oct 2017 23:55:33 -0700
Subject: v4.14-rc5 NVMeOF regression?
In-Reply-To: <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>
References: <1508192588.2493.60.camel@wdc.com>
 <82de6177-27f5-463e-3446-0305fd9f6790@grimberg.me>
 <5fb38923-36f7-c069-5f1d-96f4a9c98248@wdc.com>
 <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>
Message-ID: <20171018065533.GA7694@infradead.org>

On Wed, Oct 18, 2017 at 08:26:05AM +0300, Sagi Grimberg wrote:
> > Additionally, can you explain to me why you think the above patch is
> > necessary? As far as I can see nvme_rdma_ctrl_mutex is not held around
> > the ib_register_client(&nvme_rdma_ib_client) call nor around the
> > ib_unregister_client(&nvme_rdma_ib_client) calls.
> 
> Its held under ib_unregister_client, my understanding is that the
> _nested annotation is required for the inner mutex lock acquire.

No.  Anytime we take a single lock out of order it is a bug an a
potential deadlock.  _nested only works if we have multiple instances
of a lock (say in each inode) and we have a well defined ordering
between the instances (e.g. by the address of the inode structure).


From hch at infradead.org  Wed Oct 18 00:23:43 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 18 Oct 2017 00:23:43 -0700
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
In-Reply-To: <65999d52-2f65-2d9c-4cc4-db9d17fd9cab@broadcom.com>
References: <20171016151820.14766-1-jsmart2021@gmail.com>
 <20171017064306.GA16133@infradead.org>
 <65999d52-2f65-2d9c-4cc4-db9d17fd9cab@broadcom.com>
Message-ID: <20171018072343.GA13666@infradead.org>


On Tue, Oct 17, 2017 at 07:19:30AM -0700, James Smart wrote:
> On 10/16/2017 11:43 PM, Christoph Hellwig wrote:
> > >   	if (status)
> > >   		nvmet_set_status(req, status);
> > > +	spin_lock_irqsave(&req->sq->sqhd_lock, flags);
> > >   	if (req->sq->size)
> > >   		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
> > >   	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
> > > +	spin_unlock_irqrestore(&req->sq->sqhd_lock, flags);
> > What performance impact does this have?   I'm really reluctant to put
> > an irq disabling spinlock into a hot path for a feature that
> > theoretically is in the spec but ignored by every host.
> > 
> > I'd much rather play games with cmpxchg or similar here.
> 
> It's not ignored by every host. Its ignored by the linux host. There are
> other hosts out there that follow the spec and pay attention to sqhd - and
> the linux target would like to be used with them.
> 
> I'm open to other alternatives, but in the past, with pci adapter interfaces
> with similar "ring" index updates, the increment followed by modulo has
> always proven to be an issue for such tricks.

Something like:

	do {
		old_sqhd = req->sq->sqhd;
		new_sqhd = (old + 1) % req->sq->size;
	} while (cmpxchg(&req->sq->sqhd, old_sqhd, new_sqhd) != old_sqhd);

should do the job.


From nitzanc at mellanox.com  Wed Oct 18 00:44:09 2017
From: nitzanc at mellanox.com (Nitzan Carmi)
Date: Wed, 18 Oct 2017 07:44:09 +0000
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
Message-ID: <20171018074409.7483-1-nitzanc@mellanox.com>

Signed-off-by: Nitzan Carmi <nitzanc at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/host/rdma.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..00c033e 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1582,6 +1582,10 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
 {
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);

+	dev_dbg_ratelimited(req->queue->ctrl->ctrl.device,
+						"Reached timeout - cid %d qid %d", rq->tag,
+						cpu_to_le16(nvme_rdma_queue_idx(req->queue)));
+
 	/* queue error recovery */
 	nvme_rdma_error_recovery(req->queue->ctrl);

--
2.9.4



From hch at infradead.org  Wed Oct 18 00:45:23 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Wed, 18 Oct 2017 00:45:23 -0700
Subject: How to improve long suspend time in `nvme_shutdown_ctrl`
In-Reply-To: <72b666c9-dd57-ef87-e89e-ba280f34e90c@molgen.mpg.de>
References: <72b666c9-dd57-ef87-e89e-ba280f34e90c@molgen.mpg.de>
Message-ID: <20171018074523.GA20872@infradead.org>

On Tue, Oct 17, 2017 at 10:33:56PM +0200, Paul Menzel wrote:
> Is that the right register to read? Can this be improved? If not, is that a
> firmware bug?

Take a look at nvme_shutdown_ctrl - it's pretty simple.

It asks the device to shut down, and then polls the controller status
register every 10 milliseconds if the shutdown has completed.

So the drive performs as expected, but it is indeed rather slow.


From jthumshirn at suse.de  Wed Oct 18 01:26:47 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 18 Oct 2017 10:26:47 +0200
Subject: [PATCH v3 1/5] nvme core: allow controller RESETTING to
 RECONNECTING transition
In-Reply-To: <20171017233248.6769-2-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-2-jsmart2021@gmail.com>
Message-ID: <20171018082647.atmgy3hfarzeml3x@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 01:27:18 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 18 Oct 2017 10:27:18 +0200
Subject: [PATCH v3 2/5] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20171017233248.6769-3-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-3-jsmart2021@gmail.com>
Message-ID: <20171018082718.owx425jm642xu4jc@linux-x5ow.site>


Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 01:28:38 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 18 Oct 2017 10:28:38 +0200
Subject: [PATCH v3 3/5] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <20171017233248.6769-4-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-4-jsmart2021@gmail.com>
Message-ID: <20171018082838.vc3teuks3uf53e3x@linux-x5ow.site>

Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 01:29:29 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 18 Oct 2017 10:29:29 +0200
Subject: [PATCH v3 4/5] nvme_fc: check connectivity before initiating
 reconnects
In-Reply-To: <20171017233248.6769-5-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-5-jsmart2021@gmail.com>
Message-ID: <20171018082929.oybrk3rmzfhor6ab@linux-x5ow.site>


Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Wed Oct 18 03:12:30 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 12:12:30 +0200
Subject: [PATCH 5/9] nvme: track subsystems
In-Reply-To: <1535c6f9-8ddf-ab30-2bbd-5d9ce626b913@grimberg.me>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-6-hch@lst.de>
 <1535c6f9-8ddf-ab30-2bbd-5d9ce626b913@grimberg.me>
Message-ID: <20171018101230.GA14859@lst.de>

>>   static void nvme_free_ctrl(struct kref *kref)
>>   {
>>   	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
>> +	struct nvme_subsystem *subsys = ctrl->subsys;
>>     	put_device(ctrl->device);
>>   	nvme_release_instance(ctrl);
>>   	ida_destroy(&ctrl->ns_ida);
>>   +	if (subsys) {
>> +		mutex_lock(&subsys->lock);
>> +		list_del(&ctrl->subsys_entry);
>> +		mutex_unlock(&subsys->lock);
>> +	}
>> +
>
> When can this happen? can a controller not have a subsys?

When we fail early enough to not have set up the subsystem yet.


From hch at lst.de  Wed Oct 18 03:15:52 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 12:15:52 +0200
Subject: [PATCH 7/9] nvme: track shared namespaces
In-Reply-To: <68479c83-cdaa-77e0-232e-45c94b704765@grimberg.me>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-8-hch@lst.de>
 <68479c83-cdaa-77e0-232e-45c94b704765@grimberg.me>
Message-ID: <20171018101552.GB14859@lst.de>

On Wed, Oct 11, 2017 at 03:04:14PM +0300, Sagi Grimberg wrote:
>
>> +/*
>> + * Anchor structure for namespaces.  There is one for each namespace in a
>> + * NVMe subsystem that any of our controllers can see, and the namespace
>> + * structure for each controller is chained of it.  For private namespaces
>> + * there is a 1:1 relation to our namespace structures, that is ->list
>> + * only ever has a single entry for private namespaces.
>> + */
>> +struct nvme_ns_head {
>> +	struct list_head	list;
>
> Didn't we agree to call this list_head siblings?

Don't think I did.

>>     	struct nvme_ctrl *ctrl;
>>   	struct request_queue *queue;
>>   	struct gendisk *disk;
>> +	struct list_head siblings;
>
> And this list_head list (or entry, or sibling, or sibling_entry)?

I think siblings fits perfectly fine.  It is a lot more descriptive
than entry..


From sagi at grimberg.me  Wed Oct 18 03:44:32 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 13:44:32 +0300
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
In-Reply-To: <20171018074409.7483-1-nitzanc@mellanox.com>
References: <20171018074409.7483-1-nitzanc@mellanox.com>
Message-ID: <06c1a240-c459-6db9-e8f7-2213ec052b36@grimberg.me>

> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 92a03ff..00c033e 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -1582,6 +1582,10 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
>   {
>   	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
> 
> +	dev_dbg_ratelimited(req->queue->ctrl->ctrl.device,
> +						"Reached timeout - cid %d qid %d", rq->tag,
> +						cpu_to_le16(nvme_rdma_queue_idx(req->queue)));
> +

The alignment needs fixing.


From sagi at grimberg.me  Wed Oct 18 03:46:07 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 13:46:07 +0300
Subject: [PATCH v2] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
Message-ID: <1508323567-8536-1-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

In case we disable namespaces which has the nsid like
subsystem max_nsid we need to search for the next largest nsid
in this subsystem. If the subsystem don't has more namespaces
we set it to 0, else we take nsid from the last namespace in
namespaces list because the list is sorted while inserting.

Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
---
Changes from v1:
- Fixed compilation glitch
- moved max_nsid to a dedicated function

 drivers/nvme/target/core.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 1b208beeef50..794a3d322cd5 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -57,6 +57,18 @@ u16 nvmet_copy_from_sgl(struct nvmet_req *req, off_t off, void *buf, size_t len)
 	return 0;
 }
 
+static void nvmet_subsys_update_max_nsid(struct nvmet_subsys *subsys)
+{
+	if (list_empty(&subsys->namespaces)) {
+		subsys->max_nsid = 0;
+	} else {
+		struct nvmet_ns *ns =
+			list_last_entry(&subsys->namespaces, struct nvmet_ns, dev_link);
+
+		subsys->max_nsid = ns->nsid;
+	}
+}
+
 static u32 nvmet_async_event_result(struct nvmet_async_event *aen)
 {
 	return aen->event_type | (aen->event_info << 8) | (aen->log_page << 16);
@@ -334,6 +346,10 @@ void nvmet_ns_disable(struct nvmet_ns *ns)
 
 	ns->enabled = false;
 	list_del_rcu(&ns->dev_link);
+
+	if (ns->nsid == subsys->max_nsid)
+		nvmet_subsys_update_max_nsid(subsys);
+
 	mutex_unlock(&subsys->lock);
 
 	/*
-- 
2.7.4



From pmenzel at molgen.mpg.de  Wed Oct 18 03:58:04 2017
From: pmenzel at molgen.mpg.de (Paul Menzel)
Date: Wed, 18 Oct 2017 12:58:04 +0200
Subject: How to improve long suspend time in `nvme_shutdown_ctrl`
In-Reply-To: <20171018074523.GA20872@infradead.org>
References: <72b666c9-dd57-ef87-e89e-ba280f34e90c@molgen.mpg.de>
 <20171018074523.GA20872@infradead.org>
Message-ID: <e6cf0383-038f-b8e5-22e8-0956e04f0edd@molgen.mpg.de>

Dear Christoph,


Am 18.10.2017 um 09:45 schrieb Christoph Hellwig:
> On Tue, Oct 17, 2017 at 10:33:56PM +0200, Paul Menzel wrote:
>> Is that the right register to read? Can this be improved? If not, is that a
>> firmware bug?
> 
> Take a look at nvme_shutdown_ctrl - it's pretty simple.
> 
> It asks the device to shut down, and then polls the controller status
> register every 10 milliseconds if the shutdown has completed.
> 
> So the drive performs as expected, but it is indeed rather slow.

Thank you. Then for the record, here is the device in question. 
(Hopefully that?s the culprit.) I?ll contact TUXEDO if they can get in 
touch with Samsung.

```
$ sudo nvme id-ctrl /dev/nvme0n1
NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S316NA0HA02167
mn      : SAMSUNG MZVKW512HMJP-00000
fr      : CXA7300Q
rab     : 2
ieee    : 002538
cmic    : 0
mdts    : 0
cntlid  : 2
ver     : 10200
rtd3r   : 186a0
rtd3e   : 4c4b40
oaes    : 0
oacs    : 0x17
acl     : 7
aerl    : 3
frmw    : 0x16
lpa     : 0x3
elpe    : 63
npss    : 4
avscc   : 0x1
apsta   : 0x1
wctemp  : 343
cctemp  : 346
mtfa    : 0
hmpre   : 0
hmmin   : 0
tnvmcap : 512110190592
unvmcap : 0
rpmbs   : 0
sqes    : 0x66
cqes    : 0x44
nn      : 1
oncs    : 0x1f
fuses   : 0
fna     : 0
vwc     : 0x1
awun    : 255
awupf   : 0
nvscc   : 1
acwu    : 0
sgls    : 0
subnqn  :
ps    0 : mp:6.80W operational enlat:0 exlat:0 rrt:0 rrl:0
           rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:5.60W operational enlat:0 exlat:0 rrt:1 rrl:1
           rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:4.40W operational enlat:0 exlat:0 rrt:2 rrl:2
           rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0400W non-operational enlat:210 exlat:1500 rrt:3 rrl:3
           rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2200 exlat:6000 rrt:4 rrl:4
           rwt:4 rwl:4 idle_power:- active_power:-
```


Kind regards,

Paul


From sagi at grimberg.me  Wed Oct 18 04:07:39 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:39 +0300
Subject: [PATCH 0/5] Small rework of nvme_init/uninit_ctrl
Message-ID: <1508324864-23087-1-git-send-email-sagi@grimberg.me>

The fact that nvme_free_ctrl cleans up after nvme_init_ctrl istead
of nvme_uninit_ctrl is causing us to rely on non-trivial ctrl
refcounting inside the fabrics ->create_ctrl handlers.

Instead, have nvme_uninit_ctrl cleanup nvme_init_ctrl and fixup
rdma and loop ->create_ctrl error sequence to be sane.

This will also help when we will consolidate more code to our
common nvme-core.

nvme-pci does not seem to rely on the former behavior and thus does
not need to be fixed afaict. It seems that nvme-fc always prefers to
use the ctrl refcount scheme to cleanup the controller (also from
->create_ctrl) but it always calls nvme_uninit_ctrl before the
final put so looks like it should work as well.

Tested rdma, loop and pci but didn't test fc.

Roy Shterman (5):
  nvme-core: Make nvme_init/uninit_ctrl setup/teardown symmetric
  nvme-rdma: rework create_ctrl error flow
  nvme-rdma: remove redundant check if the ctrl is in the ctrl when
    freeing it
  nvme-loop: rework create_ctrl error flow
  nvme-loop: remove redundant check if the ctrl is in the ctrl when
    freeing it

 drivers/nvme/host/core.c   | 11 +++++------
 drivers/nvme/host/rdma.c   | 28 +++++++++++-----------------
 drivers/nvme/target/loop.c | 30 ++++++++++++------------------
 3 files changed, 28 insertions(+), 41 deletions(-)

-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 04:07:40 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:40 +0300
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508324864-23087-2-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

Its strange to have some of the cleanup for nvme_init_ctrl in
nvme_free_ctrl. This cause us to have some magic ctrl refcount
handling in controller initialization error paths.

Make the init/uninit symmetric so we can have a sane and
maintainable error sequences.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c95155696741..b8abd016202e 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2749,11 +2749,14 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
-	device_destroy(nvme_class, MKDEV(nvme_char_major, ctrl->instance));
-
 	spin_lock(&dev_list_lock);
 	list_del(&ctrl->node);
 	spin_unlock(&dev_list_lock);
+
+	ida_destroy(&ctrl->ns_ida);
+	put_device(ctrl->device);
+	device_destroy(nvme_class, MKDEV(nvme_char_major, ctrl->instance));
+	nvme_release_instance(ctrl);
 }
 EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
 
@@ -2761,10 +2764,6 @@ static void nvme_free_ctrl(struct kref *kref)
 {
 	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
 
-	put_device(ctrl->device);
-	nvme_release_instance(ctrl);
-	ida_destroy(&ctrl->ns_ida);
-
 	ctrl->ops->free_ctrl(ctrl);
 }
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 04:07:41 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:41 +0300
Subject: [PATCH 2/5] nvme-rdma: rework create_ctrl error flow
In-Reply-To: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508324864-23087-3-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

The magic ctrl refcount handling we have due to the asymmetric
behavior in nvme_init/uninit_ctrl is making the controller
creation error sequence a bit awkward and error prone as we
rely on magic refcounting to invoke ->free_ctrl from create_ctrl
error sequence.

Now that we have nvme_init/uninit_ctrl doing the right thing, we
can fix this by making the error sequence sane and simple (just
undo what we did until the error).

Also, allocate ctrl->queues before calling nvme_init_ctrl, just
to separate the driver private resource allocation and the core
resource allocations.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 605586b88186..866abee2374d 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1875,11 +1875,6 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 		}
 	}
 
-	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops,
-				0 /* no quirks, we're perfect! */);
-	if (ret)
-		goto out_free_ctrl;
-
 	INIT_DELAYED_WORK(&ctrl->reconnect_work,
 			nvme_rdma_reconnect_ctrl_work);
 	INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work);
@@ -1894,11 +1889,16 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	ctrl->queues = kcalloc(ctrl->ctrl.queue_count, sizeof(*ctrl->queues),
 				GFP_KERNEL);
 	if (!ctrl->queues)
-		goto out_uninit_ctrl;
+		goto out_free_ctrl;
+
+	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops,
+				0 /* no quirks, we're perfect! */);
+	if (ret)
+		goto out_free_queues;
 
 	ret = nvme_rdma_configure_admin_queue(ctrl, true);
 	if (ret)
-		goto out_kfree_queues;
+		goto out_uninit_ctrl;
 
 	/* sanity check icdoff */
 	if (ctrl->ctrl.icdoff) {
@@ -1954,16 +1954,14 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 
 out_remove_admin_queue:
 	nvme_rdma_destroy_admin_queue(ctrl, true);
-out_kfree_queues:
-	kfree(ctrl->queues);
 out_uninit_ctrl:
 	nvme_uninit_ctrl(&ctrl->ctrl);
-	nvme_put_ctrl(&ctrl->ctrl);
-	if (ret > 0)
-		ret = -EIO;
-	return ERR_PTR(ret);
+out_free_queues:
+	kfree(ctrl->queues);
 out_free_ctrl:
 	kfree(ctrl);
+	if (ret > 0)
+		ret = -EIO;
 	return ERR_PTR(ret);
 }
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 04:07:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:42 +0300
Subject: [PATCH 3/5] nvme-rdma: remove redundant check if the ctrl is in the
 ctrl when freeing it
In-Reply-To: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508324864-23087-4-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

Its no longer acceptable now that we do not invoke ->free_ctrl
from inside ->create_ctrl.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 866abee2374d..03a3f0372450 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -875,16 +875,12 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl)
 {
 	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
 
-	if (list_empty(&ctrl->list))
-		goto free_ctrl;
-
 	mutex_lock(&nvme_rdma_ctrl_mutex);
 	list_del(&ctrl->list);
 	mutex_unlock(&nvme_rdma_ctrl_mutex);
 
 	kfree(ctrl->queues);
 	nvmf_free_options(nctrl->opts);
-free_ctrl:
 	kfree(ctrl);
 }
 
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 04:07:43 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:43 +0300
Subject: [PATCH 4/5] nvme-loop: rework create_ctrl error flow
In-Reply-To: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508324864-23087-5-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

The magic ctrl refcount handling we have due to the asymmetric
behavior in nvme_init/uninit_ctrl is making the controller
creation error sequence a bit awkward and error prone as we
rely on magic refcounting to invoke ->free_ctrl from create_ctrl
error sequence.

Now that we have nvme_init/uninit_ctrl doing the right thing, we
can fix this by making the error sequence sane and simple (just
undo what we did until the error).

Also, allocate ctrl->queues before calling nvme_init_ctrl, just
to separate the driver private resource allocation and the core
resource allocations.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/loop.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 92628c432926..cea6bab1e70c 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -602,26 +602,24 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 
 	INIT_WORK(&ctrl->delete_work, nvme_loop_del_ctrl_work);
 	INIT_WORK(&ctrl->ctrl.reset_work, nvme_loop_reset_ctrl_work);
-
-	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_loop_ctrl_ops,
-				0 /* no quirks, we're perfect! */);
-	if (ret)
-		goto out_put_ctrl;
-
-	ret = -ENOMEM;
-
 	ctrl->ctrl.sqsize = opts->queue_size - 1;
 	ctrl->ctrl.kato = opts->kato;
 
+	ret = -ENOMEM;
 	ctrl->queues = kcalloc(opts->nr_io_queues + 1, sizeof(*ctrl->queues),
 			GFP_KERNEL);
 	if (!ctrl->queues)
-		goto out_uninit_ctrl;
+		goto out_free_ctrl;
 
-	ret = nvme_loop_configure_admin_queue(ctrl);
+	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_loop_ctrl_ops,
+				0 /* no quirks, we're perfect! */);
 	if (ret)
 		goto out_free_queues;
 
+	ret = nvme_loop_configure_admin_queue(ctrl);
+	if (ret)
+		goto out_uninit_ctrl;
+
 	if (opts->queue_size > ctrl->ctrl.maxcmd) {
 		/* warn if maxcmd is lower than queue_size */
 		dev_warn(ctrl->ctrl.device,
@@ -656,12 +654,12 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 
 out_remove_admin_queue:
 	nvme_loop_destroy_admin_queue(ctrl);
-out_free_queues:
-	kfree(ctrl->queues);
 out_uninit_ctrl:
 	nvme_uninit_ctrl(&ctrl->ctrl);
-out_put_ctrl:
-	nvme_put_ctrl(&ctrl->ctrl);
+out_free_queues:
+	kfree(ctrl->queues);
+out_free_ctrl:
+	kfree(ctrl);
 	if (ret > 0)
 		ret = -EIO;
 	return ERR_PTR(ret);
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 04:07:44 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:07:44 +0300
Subject: [PATCH 5/5] nvme-loop: remove redundant check if the ctrl is in the
 ctrl when freeing it
In-Reply-To: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508324864-23087-6-git-send-email-sagi@grimberg.me>

From: Roy Shterman <roys at lightbitslabs.com>

Its no longer acceptable now that we do not invoke ->free_ctrl
from inside ->create_ctrl.

Signed-off-by: Roy Shterman <roys at lightbitslabs.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/loop.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index cea6bab1e70c..6f9748a0b7cd 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -284,9 +284,6 @@ static void nvme_loop_free_ctrl(struct nvme_ctrl *nctrl)
 {
 	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(nctrl);
 
-	if (list_empty(&ctrl->list))
-		goto free_ctrl;
-
 	mutex_lock(&nvme_loop_ctrl_mutex);
 	list_del(&ctrl->list);
 	mutex_unlock(&nvme_loop_ctrl_mutex);
@@ -297,7 +294,6 @@ static void nvme_loop_free_ctrl(struct nvme_ctrl *nctrl)
 	}
 	kfree(ctrl->queues);
 	nvmf_free_options(nctrl->opts);
-free_ctrl:
 	kfree(ctrl);
 }
 
-- 
2.7.4



From jthumshirn at suse.de  Wed Oct 18 04:12:31 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 18 Oct 2017 13:12:31 +0200
Subject: [PATCH v2] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1508323567-8536-1-git-send-email-sagi@grimberg.me> (Sagi
 Grimberg's message of "Wed, 18 Oct 2017 13:46:07 +0300")
References: <1508323567-8536-1-git-send-email-sagi@grimberg.me>
Message-ID: <mqdvajciuao.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Wed Oct 18 04:16:24 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 14:16:24 +0300
Subject: [PATCH 5/9] nvme: track subsystems
In-Reply-To: <20171018101230.GA14859@lst.de>
References: <20170925134031.10548-1-hch@lst.de>
 <20170925134031.10548-6-hch@lst.de>
 <1535c6f9-8ddf-ab30-2bbd-5d9ce626b913@grimberg.me>
 <20171018101230.GA14859@lst.de>
Message-ID: <47322793-9d9c-abc6-e739-ecda91353ea3@grimberg.me>


>>>    static void nvme_free_ctrl(struct kref *kref)
>>>    {
>>>    	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
>>> +	struct nvme_subsystem *subsys = ctrl->subsys;
>>>      	put_device(ctrl->device);
>>>    	nvme_release_instance(ctrl);
>>>    	ida_destroy(&ctrl->ns_ida);
>>>    +	if (subsys) {
>>> +		mutex_lock(&subsys->lock);
>>> +		list_del(&ctrl->subsys_entry);
>>> +		mutex_unlock(&subsys->lock);
>>> +	}
>>> +
>>
>> When can this happen? can a controller not have a subsys?
> 
> When we fail early enough to not have set up the subsystem yet.

I think that the fact the nvme_free_ctrl cleans up after everything
that is setup in a random place is not a very good approach (just sent
a patchset that moves the uninit_ctrl cleanup back to its natural
place).

However here, given that each individual driver calls nvme_init_identify
on its own as part of its private admin configuration I don't see how
we can avoid it for now...

Hopefully we can sort this out when we centralize some more
setup/teardown logic and this subsys detach would move to where
we teardown the admin_queue (as it was added when we configured it)...


From roy.shterman at gmail.com  Wed Oct 18 04:17:26 2017
From: roy.shterman at gmail.com (roy)
Date: Wed, 18 Oct 2017 14:17:26 +0300
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
In-Reply-To: <20171018074409.7483-1-nitzanc@mellanox.com>
References: <20171018074409.7483-1-nitzanc@mellanox.com>
Message-ID: <3ba18b22-755f-0c72-9432-b6a2c09f712d@gmail.com>

On 18/10/17 10:44, Nitzan Carmi wrote:
> Signed-off-by: Nitzan Carmi <nitzanc at mellanox.com>
> Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
> ---
>   drivers/nvme/host/rdma.c | 4 ++++
>   1 file changed, 4 insertions(+)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 92a03ff..00c033e 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -1582,6 +1582,10 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
>   {
>   	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
>
> +	dev_dbg_ratelimited(req->queue->ctrl->ctrl.device,
> +						"Reached timeout - cid %d qid %d", rq->tag,
> +						cpu_to_le16(nvme_rdma_queue_idx(req->queue)));
I guess cid is command id, I think it will be better to print the 
nvme_command attributes.
something like

struct nvme_command *cmd = req->sqe.data;

and to print
cmd->common.command_id, cmd->common.opcode and the queue_id like you 
printed.

Also why printing queue idx in little Indian format?
> +
>   	/* queue error recovery */
>   	nvme_rdma_error_recovery(req->queue->ctrl);
>
> --
> 2.9.4
>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme



From israelr at mellanox.com  Wed Oct 18 05:38:26 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 12:38:26 +0000
Subject: [PATCH 3/3] nvme-loop: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <1508330306-26844-4-git-send-email-israelr@mellanox.com>

This flag is useful for admin queues that aren't used for normal IO.

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 92628c4..c56354e 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -365,6 +365,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	ctrl->admin_tag_set.driver_data = ctrl;
 	ctrl->admin_tag_set.nr_hw_queues = 1;
 	ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT;
+	ctrl->admin_tag_set.flags = BLK_MQ_F_NO_SCHED;
 
 	ctrl->queues[0].ctrl = ctrl;
 	error = nvmet_sq_init(&ctrl->queues[0].nvme_sq);
-- 
1.8.3.1



From israelr at mellanox.com  Wed Oct 18 05:38:23 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 12:38:23 +0000
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
Message-ID: <1508330306-26844-1-git-send-email-israelr@mellanox.com>

Since commit b86dd81
"block: get rid of blk-mq default scheduler choice Kconfig entries",
when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
This flag is useful for admin queues that aren't used for normal IO.
We already use this flag at nvme-pci.

Israel Rukshin (3):
  nvme-rdma: Add BLK_MQ_F_NO_SCHED flag to admin tag set
  nvme-fc: Add BLK_MQ_F_NO_SCHED flag to admin tag set
  nvme-loop: Add BLK_MQ_F_NO_SCHED flag to admin tag set

 drivers/nvme/host/fc.c     | 1 +
 drivers/nvme/host/rdma.c   | 1 +
 drivers/nvme/target/loop.c | 1 +
 3 files changed, 3 insertions(+)

-- 
1.8.3.1



From israelr at mellanox.com  Wed Oct 18 05:38:25 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 12:38:25 +0000
Subject: [PATCH 2/3] nvme-fc: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <1508330306-26844-3-git-send-email-israelr@mellanox.com>

Since commit b86dd81
"block: get rid of blk-mq default scheduler choice Kconfig entries",
when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
This flag is useful for admin queues that aren't used for normal IO.

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
---
 drivers/nvme/host/fc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index af075e9..b383735 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2796,6 +2796,7 @@ enum {
 	ctrl->admin_tag_set.driver_data = ctrl;
 	ctrl->admin_tag_set.nr_hw_queues = 1;
 	ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT;
+	ctrl->admin_tag_set.flags = BLK_MQ_F_NO_SCHED;
 
 	ret = blk_mq_alloc_tag_set(&ctrl->admin_tag_set);
 	if (ret)
-- 
1.8.3.1



From israelr at mellanox.com  Wed Oct 18 05:38:24 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 12:38:24 +0000
Subject: [PATCH 1/3] nvme-rdma: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <1508330306-26844-2-git-send-email-israelr@mellanox.com>

Since commit b86dd81
"block: get rid of blk-mq default scheduler choice Kconfig entries",
when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
This flag is useful for admin queues that aren't used for normal IO.

Signed-off-by: Israel Rukshin <israelr at mellanox.com>
---
 drivers/nvme/host/rdma.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..ee61b98f 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -699,6 +699,7 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
 		set->driver_data = ctrl;
 		set->nr_hw_queues = 1;
 		set->timeout = ADMIN_TIMEOUT;
+		set->flags = BLK_MQ_F_NO_SCHED;
 	} else {
 		set = &ctrl->tag_set;
 		memset(set, 0, sizeof(*set));
-- 
1.8.3.1



From sagi at grimberg.me  Wed Oct 18 05:50:43 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 15:50:43 +0300
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <ac3c9220-a7be-cbd4-324a-ceeb4c31c139@grimberg.me>


> Since commit b86dd81
> "block: get rid of blk-mq default scheduler choice Kconfig entries",
> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
> This flag is useful for admin queues that aren't used for normal IO.
> We already use this flag at nvme-pci.

How is the fact that we set a single hw queue has anything to do with
the admin tagset?


From dn3108 at gmail.com  Wed Oct 18 06:56:09 2017
From: dn3108 at gmail.com (Minwoo Im)
Date: Wed, 18 Oct 2017 22:56:09 +0900
Subject: [PATCH] nvme-pci: fix typos in comments
Message-ID: <1508334969-16445-1-git-send-email-dn3108@gmail.com>

fixed comment typos in adapter_alloc_cq() and adapter_alloc_sq().
'the the' duplications are replaced with 'that the'.

Signed-off-by: Minwoo Im <dn3108 at gmail.com>
---
 drivers/nvme/host/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3f5a04c..5a055c8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -930,7 +930,7 @@ static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid,
 	int flags = NVME_QUEUE_PHYS_CONTIG | NVME_CQ_IRQ_ENABLED;
 
 	/*
-	 * Note: we (ab)use the fact the the prp fields survive if no data
+	 * Note: we (ab)use the fact that the prp fields survive if no data
 	 * is attached to the request.
 	 */
 	memset(&c, 0, sizeof(c));
@@ -951,7 +951,7 @@ static int adapter_alloc_sq(struct nvme_dev *dev, u16 qid,
 	int flags = NVME_QUEUE_PHYS_CONTIG;
 
 	/*
-	 * Note: we (ab)use the fact the the prp fields survive if no data
+	 * Note: we (ab)use the fact that the prp fields survive if no data
 	 * is attached to the request.
 	 */
 	memset(&c, 0, sizeof(c));
-- 
2.7.4



From israelr at mellanox.com  Wed Oct 18 06:58:08 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 16:58:08 +0300
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <ac3c9220-a7be-cbd4-324a-ceeb4c31c139@grimberg.me>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <ac3c9220-a7be-cbd4-324a-ceeb4c31c139@grimberg.me>
Message-ID: <4f73e090-513c-aa07-a4aa-fcea4bc944a0@mellanox.com>

On 10/18/2017 3:50 PM, Sagi Grimberg wrote:
>
>> Since commit b86dd81
>> "block: get rid of blk-mq default scheduler choice Kconfig entries",
>> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline 
>> scheduler.
>> This flag is useful for admin queues that aren't used for normal IO.
>> We already use this flag at nvme-pci.
>
> How is the fact that we set a single hw queue has anything to do with
> the admin tagset?

Setting nr_hw_queues attribute of admin tag set to 1 influence the value
of nr_hw_queues attribute of the admin queue.
At /block/elevator.c
elevator_init() selects the scheduler according to the value of 
nr_hw_queues
attribute of the admin queue.

if (q->mq_ops) {
     if (q->nr_hw_queues == 1)
         e = elevator_get("mq-deadline", false);
     if (!e)
         return 0;
...

Israel


From sagi at grimberg.me  Wed Oct 18 07:31:56 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 17:31:56 +0300
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <4f73e090-513c-aa07-a4aa-fcea4bc944a0@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <ac3c9220-a7be-cbd4-324a-ceeb4c31c139@grimberg.me>
 <4f73e090-513c-aa07-a4aa-fcea4bc944a0@mellanox.com>
Message-ID: <7c0b3811-6df7-3bb7-b527-3995fdaeccb6@grimberg.me>


>>> Since commit b86dd81
>>> "block: get rid of blk-mq default scheduler choice Kconfig entries",
>>> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline 
>>> scheduler.
>>> This flag is useful for admin queues that aren't used for normal IO.
>>> We already use this flag at nvme-pci.
>>
>> How is the fact that we set a single hw queue has anything to do with
>> the admin tagset?
> 
> Setting nr_hw_queues attribute of admin tag set to 1 influence the value
> of nr_hw_queues attribute of the admin queue.
> At /block/elevator.c
> elevator_init() selects the scheduler according to the value of 
> nr_hw_queues
> attribute of the admin queue.

Oh, I confused your meaning with setting nr_hw_queues to 1 in the IO
tagset makes a difference for the admin tagset.


From keith.busch at intel.com  Wed Oct 18 08:02:15 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 09:02:15 -0600
Subject: How to improve long suspend time in `nvme_shutdown_ctrl`
In-Reply-To: <e6cf0383-038f-b8e5-22e8-0956e04f0edd@molgen.mpg.de>
References: <72b666c9-dd57-ef87-e89e-ba280f34e90c@molgen.mpg.de>
 <20171018074523.GA20872@infradead.org>
 <e6cf0383-038f-b8e5-22e8-0956e04f0edd@molgen.mpg.de>
Message-ID: <20171018150215.GC32530@localhost.localdomain>

On Wed, Oct 18, 2017 at 12:58:04PM +0200, Paul Menzel wrote:
> rtd3e   : 4c4b40

The specification recommends time to shutdown not exceed rtd3e, which you
have here at 5 seconds, so it sounds like you're well below the device's
maximum. The 338ms your system spent in nvme is already pretty good,
it seems.


From keith.busch at intel.com  Wed Oct 18 08:08:59 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 09:08:59 -0600
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <1508324864-23087-2-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
Message-ID: <20171018150859.GD32530@localhost.localdomain>

On Wed, Oct 18, 2017 at 02:07:40PM +0300, Sagi Grimberg wrote:
> From: Roy Shterman <roys at lightbitslabs.com>
> 
> Its strange to have some of the cleanup for nvme_init_ctrl in
> nvme_free_ctrl. This cause us to have some magic ctrl refcount
> handling in controller initialization error paths.
> 
> Make the init/uninit symmetric so we can have a sane and
> maintainable error sequences.

It's not magic. It's just preventing name reuse when some management
code retains an open handle after the device was removed. Otherwise
you're going to get naming clashes when you bring up a new device with
the same name from reusing the still in-use instance.


From israelr at mellanox.com  Wed Oct 18 08:16:17 2017
From: israelr at mellanox.com (Israel Rukshin)
Date: Wed, 18 Oct 2017 18:16:17 +0300
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <7c0b3811-6df7-3bb7-b527-3995fdaeccb6@grimberg.me>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <ac3c9220-a7be-cbd4-324a-ceeb4c31c139@grimberg.me>
 <4f73e090-513c-aa07-a4aa-fcea4bc944a0@mellanox.com>
 <7c0b3811-6df7-3bb7-b527-3995fdaeccb6@grimberg.me>
Message-ID: <5c549412-dd4b-4c96-97aa-64350bd21e98@mellanox.com>



On 10/18/2017 5:31 PM, Sagi Grimberg wrote:
>
>>>> Since commit b86dd81
>>>> "block: get rid of blk-mq default scheduler choice Kconfig entries",
>>>> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline 
>>>> scheduler.
>>>> This flag is useful for admin queues that aren't used for normal IO.
>>>> We already use this flag at nvme-pci.
>>>
>>> How is the fact that we set a single hw queue has anything to do with
>>> the admin tagset?
>>
>> Setting nr_hw_queues attribute of admin tag set to 1 influence the value
>> of nr_hw_queues attribute of the admin queue.
>> At /block/elevator.c
>> elevator_init() selects the scheduler according to the value of 
>> nr_hw_queues
>> attribute of the admin queue.
>
> Oh, I confused your meaning with setting nr_hw_queues to 1 in the IO
> tagset makes a difference for the admin tagset.

No,
I just wanted to say that before commit b86dd8 there was no scheduler 
for the admin q.
After commit b86dd8 I saw that mq-deadline scheduler was selected for 
the admin q.


From sagi at grimberg.me  Wed Oct 18 08:29:40 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 18:29:40 +0300
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <20171018150859.GD32530@localhost.localdomain>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171018150859.GD32530@localhost.localdomain>
Message-ID: <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>


>> From: Roy Shterman <roys at lightbitslabs.com>
>>
>> Its strange to have some of the cleanup for nvme_init_ctrl in
>> nvme_free_ctrl. This cause us to have some magic ctrl refcount
>> handling in controller initialization error paths.
>>
>> Make the init/uninit symmetric so we can have a sane and
>> maintainable error sequences.
> 
> It's not magic. It's just preventing name reuse when some management
> code retains an open handle after the device was removed.

management code as in nvme-cli?

> Otherwise
> you're going to get naming clashes when you bring up a new device with
> the same name from reusing the still in-use instance.

Can userspace keep an open-handle after device_destroy() was invoked?

Can you explain how would I see naming clashes? I'm not sure I
understand what you are referring to.


From sagi at grimberg.me  Wed Oct 18 08:30:16 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 18:30:16 +0300
Subject: [PATCH 1/3] nvme-rdma: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-2-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <1508330306-26844-2-git-send-email-israelr@mellanox.com>
Message-ID: <c3f7ab4e-4138-7688-8550-466f7c0c1228@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 18 08:30:22 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 18:30:22 +0300
Subject: [PATCH 2/3] nvme-fc: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-3-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <1508330306-26844-3-git-send-email-israelr@mellanox.com>
Message-ID: <e87fb735-fd27-820e-84ad-628efbbe53c1@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 18 08:30:29 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 18 Oct 2017 18:30:29 +0300
Subject: [PATCH 3/3] nvme-loop: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-4-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <1508330306-26844-4-git-send-email-israelr@mellanox.com>
Message-ID: <b733818c-072a-dbd8-6e55-6ac2fa360e10@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From keith.busch at intel.com  Wed Oct 18 09:21:12 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 10:21:12 -0600
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171018150859.GD32530@localhost.localdomain>
 <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>
Message-ID: <20171018162111.GA1780@localhost.localdomain>

On Wed, Oct 18, 2017 at 06:29:40PM +0300, Sagi Grimberg wrote:
> Can you explain how would I see naming clashes? I'm not sure I
> understand what you are referring to.

It's been a while since I tested this. I used to observe -EEXIST errors
setting up the kobj's from something as simple as this:

  dd if=/dev/nvme0n1 of=/dev/null conv=noerror 2> /dev/null &
  echo 1 > /sys/class/nvme/nvme0/device/remove
  echo 1 > /sys/bus/pci/rescan

The rescan would recreate nvme0n1 that still exists from the open
handle. I'm not seeing an error today; not sure what changed, but I'm
looking into that now.


From Bart.VanAssche at wdc.com  Wed Oct 18 09:24:18 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Wed, 18 Oct 2017 16:24:18 +0000
Subject: v4.14-rc5 NVMeOF regression?
In-Reply-To: <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>
References: <1508192588.2493.60.camel@wdc.com>
 <82de6177-27f5-463e-3446-0305fd9f6790@grimberg.me>
 <5fb38923-36f7-c069-5f1d-96f4a9c98248@wdc.com>
 <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>
Message-ID: <1508343856.2540.8.camel@wdc.com>

On Wed, 2017-10-18 at 08:26 +0300, Sagi Grimberg wrote:
> If you ran into a real deadlock, did you have any other output from
> hung_task watchdog? I do not yet understand the root cause from
> lockdep info provided.
>
> Also, do you know at which test-case this happened?

Hello Sagi,

Running test case 1 should be sufficient to trigger the deadlock. SysRq-w
produced the following output:

sysrq: SysRq : Show Blocked State
  task                        PC stack   pid father
kworker/u66:2   D    0   440      2 0x80000000
Workqueue: nvme-wq nvme_rdma_del_ctrl_work [nvme_rdma]
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x221/0x580
 io_schedule_timeout+0x1e/0x50
 wait_for_completion_io_timeout+0x118/0x180
 blk_execute_rq+0x86/0xc0
 __nvme_submit_sync_cmd+0x89/0xf0
 nvmf_reg_write32+0x4b/0x90 [nvme_fabrics]
 nvme_shutdown_ctrl+0x41/0xe0
 nvme_rdma_shutdown_ctrl+0xca/0xd0 [nvme_rdma]
 nvme_rdma_remove_ctrl+0x2b/0x40 [nvme_rdma]
 nvme_rdma_del_ctrl_work+0x25/0x30 [nvme_rdma]
 process_one_work+0x1fd/0x630
 worker_thread+0x1db/0x3b0
 kthread+0x11e/0x150
 ret_from_fork+0x27/0x40
01              D    0  2868   2862 0x00000000
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x260/0x580
 wait_for_completion+0x108/0x170
 flush_work+0x1e0/0x270
 nvme_rdma_del_ctrl+0x5a/0x80 [nvme_rdma]
 nvme_sysfs_delete+0x2a/0x40
 dev_attr_store+0x18/0x30
 sysfs_kf_write+0x45/0x60
 kernfs_fop_write+0x124/0x1c0
 __vfs_write+0x28/0x150
 vfs_write+0xc7/0x1b0
 SyS_write+0x49/0xa0
 entry_SYSCALL_64_fastpath+0x18/0xad

Bart.

From jsmart2021 at gmail.com  Wed Oct 18 09:45:42 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 18 Oct 2017 09:45:42 -0700
Subject: [PATCH v2] nvmet: protect sqhd update by a lock
In-Reply-To: <20171018072343.GA13666@infradead.org>
References: <20171016151820.14766-1-jsmart2021@gmail.com>
 <20171017064306.GA16133@infradead.org>
 <65999d52-2f65-2d9c-4cc4-db9d17fd9cab@broadcom.com>
 <20171018072343.GA13666@infradead.org>
Message-ID: <3ac0e8f3-69f1-e147-6221-809d68cb1ccf@gmail.com>

On 10/18/2017 12:23 AM, Christoph Hellwig wrote:
> do {
> 		old_sqhd = req->sq->sqhd;
> 		new_sqhd = (old + 1) % req->sq->size;
> 	} while (cmpxchg(&req->sq->sqhd, old_sqhd, new_sqhd) != old_sqhd);

sweet. does look good.

I'll repost as v3, but with an ammended title

-- james



From jsmart2021 at gmail.com  Wed Oct 18 09:52:16 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 18 Oct 2017 09:52:16 -0700
Subject: [PATCH v3] nvmet: synchronize sqhd update
Message-ID: <20171018165216.25563-1-jsmart2021@gmail.com>

In testing target io in read write mix, we did indeed get into cases
where sqhd didn't update properly and slowly missed enough updates to
shutdown the queue.

Protect the updating sqhd by using cmpxchg

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2: move locks so around update and assignment
v3: convert to cmpxchg
---
 drivers/nvme/target/core.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index cfc35cbb6fe2..72d02f871c03 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -387,11 +387,18 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 static void __nvmet_req_complete(struct nvmet_req *req, u16 status)
 {
+	u16 old_sqhd, new_sqhd;
+
 	if (status)
 		nvmet_set_status(req, status);
 
-	if (req->sq->size)
-		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
+	if (req->sq->size) {
+		do {
+			old_sqhd = req->sq->sqhd;
+			new_sqhd = (old_sqhd + 1) % req->sq->size;
+		} while (cmpxchg(&req->sq->sqhd, old_sqhd, new_sqhd) !=
+					old_sqhd);
+	}
 	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
 	req->rsp->sq_id = cpu_to_le16(req->sq->qid);
 	req->rsp->command_id = req->cmd->common.command_id;
-- 
2.13.1



From hch at lst.de  Wed Oct 18 09:52:41 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:41 +0200
Subject: nvme multipath support V4
Message-ID: <20171018165258.23212-1-hch@lst.de>

Hi all,

this series adds support for multipathing, that is accessing nvme
namespaces through multiple controllers to the nvme core driver.

It is a very thin and efficient implementation that relies on
close cooperation with other bits of the nvme driver, and few small
and simple block helpers.

Compared to dm-multipath the important differences are how management
of the paths is done, and how the I/O path works.

Management of the paths is fully integrated into the nvme driver,
for each newly found nvme controller we check if there are other
controllers that refer to the same subsystem, and if so we link them
up in the nvme driver.  Then for each namespace found we check if
the namespace id and identifiers match to check if we have multiple
controllers that refer to the same namespaces.  For now path
availability is based entirely on the controller status, which at
least for fabrics will be continuously updated based on the mandatory
keep alive timer.  Once the Asynchronous Namespace Access (ANA)
proposal passes in NVMe we will also get per-namespace states in
addition to that, but for now any details of that remain confidential
to NVMe members.

The I/O path is very different from the existing multipath drivers,
which is enabled by the fact that NVMe (unlike SCSI) does not support
partial completions - a controller will either complete a whole
command or not, but never only complete parts of it.  Because of that
there is no need to clone bios or requests - the I/O path simply
redirects the I/O to a suitable path.  For successful commands
multipath is not in the completion stack at all.  For failed commands
we decide if the error could be a path failure, and if yes remove
the bios from the request structure and requeue them before completing
the request.  All together this means there is no performance
degradation compared to normal nvme operation when using the multipath
device node (at least not until I find a dual ported DRAM backed
device :))

A git tree is available at:

   git://git.infradead.org/users/hch/block.git nvme-mpath

gitweb:

   http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/nvme-mpath

Changes since V3:
  - new block layer support for hidden gendisks
  - a couple new patches to refactor device handling before the
    actual multipath support
  - don't expose per-controller block device nodes
  - use /dev/nvmeXnZ as the device nodes for the whole subsystem.
  - expose subsystems in sysfs (Hannes Reinecke)
  - fix a subsystem leak when duplicate NQNs are found
  - fix up some names
  - don't clear current_path if freeing a different namespace

Changes since V2:
  - don't create duplicate subsystems on reset (Keith Bush)
  - free requests properly when failing over in I/O completion (Keith Bush)
  - new devices names: /dev/nvm-sub%dn%d
  - expose the namespace identification sysfs files for the mpath nodes

Changes since V1:
  - introduce new nvme_ns_ids structure to clean up identifier handling
  - generic_make_request_fast is now named direct_make_request and calls
    generic_make_request_checks
  - reset bi_disk on resubmission
  - create sysfs links between the existing nvme namespace block devices and
    the new share mpath device
  - temporarily added the timeout patches from James, this should go into
    nvme-4.14, though


From hch at lst.de  Wed Oct 18 09:52:42 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:42 +0200
Subject: [PATCH 01/17] block: move REQ_NOWAIT
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-2-hch@lst.de>

This flag should be before the operation-specific REQ_NOUNMAP bit.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 include/linux/blk_types.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index a2d2aa709cef..acc2f3cdc2fc 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -224,11 +224,11 @@ enum req_flag_bits {
 	__REQ_PREFLUSH,		/* request for cache flush */
 	__REQ_RAHEAD,		/* read ahead, can fail anytime */
 	__REQ_BACKGROUND,	/* background IO */
+	__REQ_NOWAIT,           /* Don't wait if request will block */
 
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
-	__REQ_NOWAIT,           /* Don't wait if request will block */
 	__REQ_NR_BITS,		/* stops here */
 };
 
@@ -245,9 +245,9 @@ enum req_flag_bits {
 #define REQ_PREFLUSH		(1ULL << __REQ_PREFLUSH)
 #define REQ_RAHEAD		(1ULL << __REQ_RAHEAD)
 #define REQ_BACKGROUND		(1ULL << __REQ_BACKGROUND)
+#define REQ_NOWAIT		(1ULL << __REQ_NOWAIT)
 
 #define REQ_NOUNMAP		(1ULL << __REQ_NOUNMAP)
-#define REQ_NOWAIT		(1ULL << __REQ_NOWAIT)
 
 #define REQ_FAILFAST_MASK \
 	(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:43 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:43 +0200
Subject: [PATCH 02/17] block: add REQ_DRV bit
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-3-hch@lst.de>

Set aside a bit in the request/bio flags for driver use.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 include/linux/blk_types.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index acc2f3cdc2fc..7ec2ed097a8a 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -229,6 +229,9 @@ enum req_flag_bits {
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
+	/* for driver use */
+	__REQ_DRV,
+
 	__REQ_NR_BITS,		/* stops here */
 };
 
@@ -249,6 +252,8 @@ enum req_flag_bits {
 
 #define REQ_NOUNMAP		(1ULL << __REQ_NOUNMAP)
 
+#define REQ_DRV			(1ULL << __REQ_DRV)
+
 #define REQ_FAILFAST_MASK \
 	(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
 
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:44 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:44 +0200
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-4-hch@lst.de>

This helper allows reinserting a bio into a new queue without much
overhead, but requires all queue limits to be the same for the upper
and lower queues, and it does not provide any recursion preventions.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 block/blk-core.c       | 34 ++++++++++++++++++++++++++++++++++
 include/linux/blkdev.h |  1 +
 2 files changed, 35 insertions(+)

diff --git a/block/blk-core.c b/block/blk-core.c
index 14f7674fa0b1..b8c80f39f5fe 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2241,6 +2241,40 @@ blk_qc_t generic_make_request(struct bio *bio)
 }
 EXPORT_SYMBOL(generic_make_request);
 
+/**
+ * direct_make_request - hand a buffer directly to its device driver for I/O
+ * @bio:  The bio describing the location in memory and on the device.
+ *
+ * This function behaves like generic_make_request(), but does not protect
+ * against recursion.  Must only be used if the called driver is known
+ * to not call generic_make_request (or direct_make_request) again from
+ * its make_request function.  (Calling direct_make_request again from
+ * a workqueue is perfectly fine as that doesn't recurse).
+ */
+blk_qc_t direct_make_request(struct bio *bio)
+{
+	struct request_queue *q = bio->bi_disk->queue;
+	bool nowait = bio->bi_opf & REQ_NOWAIT;
+	blk_qc_t ret;
+
+	if (!generic_make_request_checks(bio))
+		return BLK_QC_T_NONE;
+
+	if (unlikely(blk_queue_enter(q, nowait))) {
+		if (nowait && !blk_queue_dying(q))
+			bio->bi_status = BLK_STS_AGAIN;
+		else
+			bio->bi_status = BLK_STS_IOERR;
+		bio_endio(bio);
+		return BLK_QC_T_NONE;
+	}
+
+	ret = q->make_request_fn(q, bio);
+	blk_queue_exit(q);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(direct_make_request);
+
 /**
  * submit_bio - submit a bio to the block device layer for I/O
  * @bio: The &struct bio which describes the I/O
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 02fa42d24b52..780f01db5899 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -936,6 +936,7 @@ do {								\
 extern int blk_register_queue(struct gendisk *disk);
 extern void blk_unregister_queue(struct gendisk *disk);
 extern blk_qc_t generic_make_request(struct bio *bio);
+extern blk_qc_t direct_make_request(struct bio *bio);
 extern void blk_rq_init(struct request_queue *q, struct request *rq);
 extern void blk_init_request_from_bio(struct request *req, struct bio *bio);
 extern void blk_put_request(struct request *);
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:45 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:45 +0200
Subject: [PATCH 04/17] block: add a blk_steal_bios helper
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-5-hch@lst.de>

This helpers allows to bounce steal the uncompleted bios from a request so
that they can be reissued on another path.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 block/blk-core.c       | 20 ++++++++++++++++++++
 include/linux/blkdev.h |  2 ++
 2 files changed, 22 insertions(+)

diff --git a/block/blk-core.c b/block/blk-core.c
index b8c80f39f5fe..e804529e65a5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2767,6 +2767,26 @@ struct request *blk_fetch_request(struct request_queue *q)
 }
 EXPORT_SYMBOL(blk_fetch_request);
 
+/*
+ * Steal bios from a request.  The request must not have been partially
+ * completed before.
+ */
+void blk_steal_bios(struct bio_list *list, struct request *rq)
+{
+	if (rq->bio) {
+		if (list->tail)
+			list->tail->bi_next = rq->bio;
+		else
+			list->head = rq->bio;
+		list->tail = rq->biotail;
+	}
+
+	rq->bio = NULL;
+	rq->biotail = NULL;
+	rq->__data_len = 0;
+}
+EXPORT_SYMBOL_GPL(blk_steal_bios);
+
 /**
  * blk_update_request - Special helper function for request stacking drivers
  * @req:      the request being processed
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 780f01db5899..45c63764a14e 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1110,6 +1110,8 @@ extern struct request *blk_peek_request(struct request_queue *q);
 extern void blk_start_request(struct request *rq);
 extern struct request *blk_fetch_request(struct request_queue *q);
 
+void blk_steal_bios(struct bio_list *list, struct request *rq);
+
 /*
  * Request completion related functions.
  *
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:46 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:46 +0200
Subject: [PATCH 05/17] block: don't look at the struct device dev_t in
 disk_devt
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-6-hch@lst.de>

The hidden gendisks introduced in the next patch need to keep the dev
field in their struct device empty so that udev won't try to create
block device nodes for them.  To support that rewrite disk_devt to
look at the major and first_minor fields in the gendisk itself instead
of looking into the struct device.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 block/genhd.c         | 4 ----
 include/linux/genhd.h | 2 +-
 2 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index dd305c65ffb0..1174d24e405e 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -649,10 +649,6 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 		return;
 	}
 	disk_to_dev(disk)->devt = devt;
-
-	/* ->major and ->first_minor aren't supposed to be
-	 * dereferenced from here on, but set them just in case.
-	 */
 	disk->major = MAJOR(devt);
 	disk->first_minor = MINOR(devt);
 
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index ea652bfcd675..5c0ed5db33c2 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -234,7 +234,7 @@ static inline bool disk_part_scan_enabled(struct gendisk *disk)
 
 static inline dev_t disk_devt(struct gendisk *disk)
 {
-	return disk_to_dev(disk)->devt;
+	return MKDEV(disk->major, disk->first_minor);
 }
 
 static inline dev_t part_devt(struct hd_struct *part)
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:47 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:47 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-7-hch@lst.de>

With this flag a driver can create a gendisk that can be used for I/O
submission inside the kernel, but which is not registered as user
facing block device.  This will be useful for the NVMe multipath
implementation.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 block/genhd.c         | 53 ++++++++++++++++++++++++++++++++++-----------------
 include/linux/genhd.h |  1 +
 2 files changed, 36 insertions(+), 18 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index 1174d24e405e..0b28cd491b1d 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -585,6 +585,11 @@ static void register_disk(struct device *parent, struct gendisk *disk)
 	 */
 	pm_runtime_set_memalloc_noio(ddev, true);
 
+	if (disk->flags & GENHD_FL_HIDDEN) {
+		dev_set_uevent_suppress(ddev, 0);
+		return;
+	}
+
 	disk->part0.holder_dir = kobject_create_and_add("holders", &ddev->kobj);
 	disk->slave_dir = kobject_create_and_add("slaves", &ddev->kobj);
 
@@ -616,6 +621,11 @@ static void register_disk(struct device *parent, struct gendisk *disk)
 	while ((part = disk_part_iter_next(&piter)))
 		kobject_uevent(&part_to_dev(part)->kobj, KOBJ_ADD);
 	disk_part_iter_exit(&piter);
+
+	err = sysfs_create_link(&ddev->kobj,
+				&disk->queue->backing_dev_info->dev->kobj,
+				"bdi");
+	WARN_ON(err);
 }
 
 /**
@@ -630,7 +640,6 @@ static void register_disk(struct device *parent, struct gendisk *disk)
  */
 void device_add_disk(struct device *parent, struct gendisk *disk)
 {
-	struct backing_dev_info *bdi;
 	dev_t devt;
 	int retval;
 
@@ -639,7 +648,8 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 	 * parameters make sense.
 	 */
 	WARN_ON(disk->minors && !(disk->major || disk->first_minor));
-	WARN_ON(!disk->minors && !(disk->flags & GENHD_FL_EXT_DEVT));
+	WARN_ON(!disk->minors &&
+		!(disk->flags & (GENHD_FL_EXT_DEVT | GENHD_FL_HIDDEN)));
 
 	disk->flags |= GENHD_FL_UP;
 
@@ -648,18 +658,26 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 		WARN_ON(1);
 		return;
 	}
-	disk_to_dev(disk)->devt = devt;
 	disk->major = MAJOR(devt);
 	disk->first_minor = MINOR(devt);
 
 	disk_alloc_events(disk);
 
-	/* Register BDI before referencing it from bdev */
-	bdi = disk->queue->backing_dev_info;
-	bdi_register_owner(bdi, disk_to_dev(disk));
-
-	blk_register_region(disk_devt(disk), disk->minors, NULL,
-			    exact_match, exact_lock, disk);
+	if (disk->flags & GENHD_FL_HIDDEN) {
+		/*
+		 * Don't let hidden disks show up in /proc/partitions,
+		 * and don't bother scanning for partitions either.
+		 */
+		disk->flags |= GENHD_FL_SUPPRESS_PARTITION_INFO;
+		disk->flags |= GENHD_FL_NO_PART_SCAN;
+	} else {
+		/* Register BDI before referencing it from bdev */
+		disk_to_dev(disk)->devt = devt;
+		bdi_register_owner(disk->queue->backing_dev_info,
+				disk_to_dev(disk));
+		blk_register_region(disk_devt(disk), disk->minors, NULL,
+				    exact_match, exact_lock, disk);
+	}
 	register_disk(parent, disk);
 	blk_register_queue(disk);
 
@@ -669,10 +687,6 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 	 */
 	WARN_ON_ONCE(!blk_get_queue(disk->queue));
 
-	retval = sysfs_create_link(&disk_to_dev(disk)->kobj, &bdi->dev->kobj,
-				   "bdi");
-	WARN_ON(retval);
-
 	disk_add_events(disk);
 	blk_integrity_add(disk);
 }
@@ -701,7 +715,8 @@ void del_gendisk(struct gendisk *disk)
 	set_capacity(disk, 0);
 	disk->flags &= ~GENHD_FL_UP;
 
-	sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
+	if (!(disk->flags & GENHD_FL_HIDDEN))
+		sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
 	if (disk->queue) {
 		/*
 		 * Unregister bdi before releasing device numbers (as they can
@@ -712,13 +727,15 @@ void del_gendisk(struct gendisk *disk)
 	} else {
 		WARN_ON(1);
 	}
-	blk_unregister_region(disk_devt(disk), disk->minors);
+
+	if (!(disk->flags & GENHD_FL_HIDDEN)) {
+		blk_unregister_region(disk_devt(disk), disk->minors);
+		kobject_put(disk->part0.holder_dir);
+		kobject_put(disk->slave_dir);
+	}
 
 	part_stat_set_all(&disk->part0, 0);
 	disk->part0.stamp = 0;
-
-	kobject_put(disk->part0.holder_dir);
-	kobject_put(disk->slave_dir);
 	if (!sysfs_deprecated)
 		sysfs_remove_link(block_depr, dev_name(disk_to_dev(disk)));
 	pm_runtime_set_memalloc_noio(disk_to_dev(disk), false);
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 5c0ed5db33c2..93aae3476f58 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -140,6 +140,7 @@ struct hd_struct {
 #define GENHD_FL_NATIVE_CAPACITY		128
 #define GENHD_FL_BLOCK_EVENTS_ON_EXCL_WRITE	256
 #define GENHD_FL_NO_PART_SCAN			512
+#define GENHD_FL_HIDDEN				1024
 
 enum {
 	DISK_EVENT_MEDIA_CHANGE			= 1 << 0, /* media changed */
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:48 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:48 +0200
Subject: [PATCH 07/17] nvme: use ida_simple_{get,
 remove} for the controller instance
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-8-hch@lst.de>

Switch to the ida_simple_* helpers instead of opencoding them.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 40 +++++++---------------------------------
 1 file changed, 7 insertions(+), 33 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 573cc3b59bfa..bdae1e4fcd0b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -74,6 +74,8 @@ EXPORT_SYMBOL_GPL(nvme_wq);
 static LIST_HEAD(nvme_ctrl_list);
 static DEFINE_SPINLOCK(dev_list_lock);
 
+static DEFINE_IDA(nvme_instance_ida);
+
 static struct class *nvme_class;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
@@ -2696,35 +2698,6 @@ void nvme_queue_async_events(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_queue_async_events);
 
-static DEFINE_IDA(nvme_instance_ida);
-
-static int nvme_set_instance(struct nvme_ctrl *ctrl)
-{
-	int instance, error;
-
-	do {
-		if (!ida_pre_get(&nvme_instance_ida, GFP_KERNEL))
-			return -ENODEV;
-
-		spin_lock(&dev_list_lock);
-		error = ida_get_new(&nvme_instance_ida, &instance);
-		spin_unlock(&dev_list_lock);
-	} while (error == -EAGAIN);
-
-	if (error)
-		return -ENODEV;
-
-	ctrl->instance = instance;
-	return 0;
-}
-
-static void nvme_release_instance(struct nvme_ctrl *ctrl)
-{
-	spin_lock(&dev_list_lock);
-	ida_remove(&nvme_instance_ida, ctrl->instance);
-	spin_unlock(&dev_list_lock);
-}
-
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_stop_keep_alive(ctrl);
@@ -2762,7 +2735,7 @@ static void nvme_free_ctrl(struct kref *kref)
 	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
 
 	put_device(ctrl->device);
-	nvme_release_instance(ctrl);
+	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 	ida_destroy(&ctrl->ns_ida);
 
 	ctrl->ops->free_ctrl(ctrl);
@@ -2796,9 +2769,10 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	INIT_WORK(&ctrl->async_event_work, nvme_async_event_work);
 	INIT_WORK(&ctrl->fw_act_work, nvme_fw_act_work);
 
-	ret = nvme_set_instance(ctrl);
-	if (ret)
+	ret = ida_simple_get(&nvme_instance_ida, 0, 0, GFP_KERNEL);
+	if (ret < 0)
 		goto out;
+	ctrl->instance = ret;
 
 	ctrl->device = device_create_with_groups(nvme_class, ctrl->dev,
 				MKDEV(nvme_char_major, ctrl->instance),
@@ -2825,7 +2799,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	return 0;
 out_release_instance:
-	nvme_release_instance(ctrl);
+	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 out:
 	return ret;
 }
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:49 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:49 +0200
Subject: [PATCH 08/17] nvme: use kref_get_unless_zero in nvme_find_get_ns
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-9-hch@lst.de>

For kref_get_unless_zero to protect against lookup vs free races we need
to use it in all places where we aren't guaranteed to already hold a
reference.  There is no such guarantee in nvme_find_get_ns, so switch to
kref_get_unless_zero in this function.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index bdae1e4fcd0b..3f51bd0326b5 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2290,7 +2290,8 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	mutex_lock(&ctrl->namespaces_mutex);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
 		if (ns->ns_id == nsid) {
-			kref_get(&ns->kref);
+			if (!kref_get_unless_zero(&ns->kref))
+				continue;
 			ret = ns;
 			break;
 		}
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:50 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:50 +0200
Subject: [PATCH 09/17] nvme: simplify nvme_open
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-10-hch@lst.de>

Now that we are protected against lookup vs free races for the namespace
by using kref_get_unless_zero we don't need the hack of NULLing out the
disk private data during removal.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 40 ++++++++++------------------------------
 1 file changed, 10 insertions(+), 30 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3f51bd0326b5..05a6fad89352 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -253,12 +253,6 @@ static void nvme_free_ns(struct kref *kref)
 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
 
-	if (ns->disk) {
-		spin_lock(&dev_list_lock);
-		ns->disk->private_data = NULL;
-		spin_unlock(&dev_list_lock);
-	}
-
 	put_disk(ns->disk);
 	ida_simple_remove(&ns->ctrl->ns_ida, ns->instance);
 	nvme_put_ctrl(ns->ctrl);
@@ -270,29 +264,6 @@ static void nvme_put_ns(struct nvme_ns *ns)
 	kref_put(&ns->kref, nvme_free_ns);
 }
 
-static struct nvme_ns *nvme_get_ns_from_disk(struct gendisk *disk)
-{
-	struct nvme_ns *ns;
-
-	spin_lock(&dev_list_lock);
-	ns = disk->private_data;
-	if (ns) {
-		if (!kref_get_unless_zero(&ns->kref))
-			goto fail;
-		if (!try_module_get(ns->ctrl->ops->module))
-			goto fail_put_ns;
-	}
-	spin_unlock(&dev_list_lock);
-
-	return ns;
-
-fail_put_ns:
-	kref_put(&ns->kref, nvme_free_ns);
-fail:
-	spin_unlock(&dev_list_lock);
-	return NULL;
-}
-
 struct request *nvme_alloc_request(struct request_queue *q,
 		struct nvme_command *cmd, unsigned int flags, int qid)
 {
@@ -1056,7 +1027,16 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 
 static int nvme_open(struct block_device *bdev, fmode_t mode)
 {
-	return nvme_get_ns_from_disk(bdev->bd_disk) ? 0 : -ENXIO;
+	struct nvme_ns *ns = bdev->bd_disk->private_data;
+
+	if (!kref_get_unless_zero(&ns->kref))
+		return -ENXIO;
+	if (!try_module_get(ns->ctrl->ops->module)) {
+		kref_put(&ns->kref, nvme_free_ns);
+		return -ENXIO;
+	}
+
+	return 0;
 }
 
 static void nvme_release(struct gendisk *disk, fmode_t mode)
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:51 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:51 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct device
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-11-hch@lst.de>

Instead of allocating a separate struct device for the character device
handle embedd it into struct nvme_ctrl and use it for the main controller
refcounting.  This removes double refcounting and gets us an automatic
reference for the character device operations.  We keep ctrl->device as a
pointer for now to avoid chaning printks all over, but in the future we
could look into message printing helpers that take a controller structure
similar to what other subsystems do.

Note the delete_ctrl operation always already has a reference when it
is entered now, so we don't need to do the unless_zero variant there.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c   | 43 ++++++++++++++++++++++---------------------
 drivers/nvme/host/fc.c     |  8 ++------
 drivers/nvme/host/nvme.h   | 12 +++++++++++-
 drivers/nvme/host/pci.c    |  2 +-
 drivers/nvme/host/rdma.c   |  5 ++---
 drivers/nvme/target/loop.c |  2 +-
 6 files changed, 39 insertions(+), 33 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 05a6fad89352..7c1d1c4f93fc 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1915,7 +1915,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file)
 			ret = -EWOULDBLOCK;
 			break;
 		}
-		if (!kref_get_unless_zero(&ctrl->kref))
+		if (!kobject_get_unless_zero(&ctrl->device->kobj))
 			break;
 		file->private_data = ctrl;
 		ret = 0;
@@ -2374,7 +2374,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	list_add_tail(&ns->list, &ctrl->namespaces);
 	mutex_unlock(&ctrl->namespaces_mutex);
 
-	kref_get(&ctrl->kref);
+	nvme_get_ctrl(ctrl);
 
 	kfree(id);
 
@@ -2703,7 +2703,7 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
-	device_destroy(nvme_class, MKDEV(nvme_char_major, ctrl->instance));
+	device_del(ctrl->device);
 
 	spin_lock(&dev_list_lock);
 	list_del(&ctrl->node);
@@ -2711,23 +2711,17 @@ void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
 
-static void nvme_free_ctrl(struct kref *kref)
+static void nvme_free_ctrl(struct device *dev)
 {
-	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
+	struct nvme_ctrl *ctrl =
+		container_of(dev, struct nvme_ctrl, ctrl_device);
 
-	put_device(ctrl->device);
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 	ida_destroy(&ctrl->ns_ida);
 
 	ctrl->ops->free_ctrl(ctrl);
 }
 
-void nvme_put_ctrl(struct nvme_ctrl *ctrl)
-{
-	kref_put(&ctrl->kref, nvme_free_ctrl);
-}
-EXPORT_SYMBOL_GPL(nvme_put_ctrl);
-
 /*
  * Initialize a NVMe controller structures.  This needs to be called during
  * earliest initialization so that we have the initialized structured around
@@ -2742,7 +2736,6 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	spin_lock_init(&ctrl->lock);
 	INIT_LIST_HEAD(&ctrl->namespaces);
 	mutex_init(&ctrl->namespaces_mutex);
-	kref_init(&ctrl->kref);
 	ctrl->dev = dev;
 	ctrl->ops = ops;
 	ctrl->quirks = quirks;
@@ -2755,15 +2748,21 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 		goto out;
 	ctrl->instance = ret;
 
-	ctrl->device = device_create_with_groups(nvme_class, ctrl->dev,
-				MKDEV(nvme_char_major, ctrl->instance),
-				ctrl, nvme_dev_attr_groups,
-				"nvme%d", ctrl->instance);
-	if (IS_ERR(ctrl->device)) {
-		ret = PTR_ERR(ctrl->device);
+	device_initialize(&ctrl->ctrl_device);
+	ctrl->device = &ctrl->ctrl_device;
+	ctrl->device->devt = MKDEV(nvme_char_major, ctrl->instance);
+	ctrl->device->class = nvme_class;
+	ctrl->device->parent = ctrl->dev;
+	ctrl->device->groups = nvme_dev_attr_groups;
+	ctrl->device->release = nvme_free_ctrl;
+	dev_set_drvdata(ctrl->device, ctrl);
+	ret = dev_set_name(ctrl->device, "nvme%d", ctrl->instance);
+	if (ret)
 		goto out_release_instance;
-	}
-	get_device(ctrl->device);
+	ret = device_add(ctrl->device);
+	if (ret)
+		goto out_free_name;
+
 	ida_init(&ctrl->ns_ida);
 
 	spin_lock(&dev_list_lock);
@@ -2779,6 +2778,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 		min(default_ps_max_latency_us, (unsigned long)S32_MAX));
 
 	return 0;
+out_free_name:
+	kfree_const(dev->kobj.name);
 out_release_instance:
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 out:
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index b8e0822127c1..e45c6350e01c 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2681,14 +2681,10 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
 	int ret;
 
-	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
-		return -EBUSY;
-
+	nvme_get_ctrl(&ctrl->ctrl);
 	ret = __nvme_fc_del_ctrl(ctrl);
-
 	if (!ret)
 		flush_workqueue(nvme_wq);
-
 	nvme_put_ctrl(&ctrl->ctrl);
 
 	return ret;
@@ -2905,7 +2901,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 		return ERR_PTR(ret);
 	}
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	dev_info(ctrl->ctrl.device,
 		"NVME-FC{%d}: new ctrl: NQN \"%s\"\n",
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index df787f39f4c1..55286acd5387 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -127,12 +127,12 @@ struct nvme_ctrl {
 	struct request_queue *admin_q;
 	struct request_queue *connect_q;
 	struct device *dev;
-	struct kref kref;
 	int instance;
 	struct blk_mq_tag_set *tagset;
 	struct blk_mq_tag_set *admin_tagset;
 	struct list_head namespaces;
 	struct mutex namespaces_mutex;
+	struct device ctrl_device;
 	struct device *device;	/* char device */
 	struct list_head node;
 	struct ida ns_ida;
@@ -278,6 +278,16 @@ static inline void nvme_end_request(struct request *req, __le16 status,
 	blk_mq_complete_request(req);
 }
 
+static inline void nvme_get_ctrl(struct nvme_ctrl *ctrl)
+{
+	get_device(ctrl->device);
+}
+
+static inline void nvme_put_ctrl(struct nvme_ctrl *ctrl)
+{
+	put_device(ctrl->device);
+}
+
 void nvme_complete_rq(struct request *req);
 void nvme_cancel_request(struct request *req, void *data, bool reserved);
 bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index cb73bc8cad3b..485f38558ba1 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2132,7 +2132,7 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev, int status)
 {
 	dev_warn(dev->ctrl.device, "Removing after probe failure status: %d\n", status);
 
-	kref_get(&dev->ctrl.kref);
+	nvme_get_ctrl(&dev->ctrl);
 	nvme_dev_disable(dev, false);
 	if (!schedule_work(&dev->remove_work))
 		nvme_put_ctrl(&dev->ctrl);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..63f649c655f0 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1793,8 +1793,7 @@ static int nvme_rdma_del_ctrl(struct nvme_ctrl *nctrl)
 	 * Keep a reference until all work is flushed since
 	 * __nvme_rdma_del_ctrl can free the ctrl mem
 	 */
-	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
-		return -EBUSY;
+	nvme_get_ctrl(&ctrl->ctrl);
 	ret = __nvme_rdma_del_ctrl(ctrl);
 	if (!ret)
 		flush_work(&ctrl->delete_work);
@@ -1950,7 +1949,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	dev_info(ctrl->ctrl.device, "new ctrl: NQN \"%s\", addr %pISpcs\n",
 		ctrl->ctrl.opts->subsysnqn, &ctrl->addr);
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	mutex_lock(&nvme_rdma_ctrl_mutex);
 	list_add_tail(&ctrl->list, &nvme_rdma_ctrl_list);
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 92628c432926..818864c33979 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -641,7 +641,7 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 	dev_info(ctrl->ctrl.device,
 		 "new ctrl: \"%s\"\n", ctrl->ctrl.opts->subsysnqn);
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
 	WARN_ON_ONCE(!changed);
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:52 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:52 +0200
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-12-hch@lst.de>

Use the core chrdev code to set up the link between the character device
and the nvme controller.  This allows us to get rid of the global list
of all controllers, and also ensures that we have both a reference to
the controller and the transport module before the open method of the
character device is called.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 76 ++++++++++--------------------------------------
 drivers/nvme/host/nvme.h |  3 +-
 2 files changed, 18 insertions(+), 61 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7c1d1c4f93fc..3757f5695de2 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -52,9 +52,6 @@ static u8 nvme_max_retries = 5;
 module_param_named(max_retries, nvme_max_retries, byte, 0644);
 MODULE_PARM_DESC(max_retries, "max number of retries a command may have");
 
-static int nvme_char_major;
-module_param(nvme_char_major, int, 0);
-
 static unsigned long default_ps_max_latency_us = 100000;
 module_param(default_ps_max_latency_us, ulong, 0644);
 MODULE_PARM_DESC(default_ps_max_latency_us,
@@ -71,11 +68,8 @@ MODULE_PARM_DESC(streams, "turn on support for Streams write directives");
 struct workqueue_struct *nvme_wq;
 EXPORT_SYMBOL_GPL(nvme_wq);
 
-static LIST_HEAD(nvme_ctrl_list);
-static DEFINE_SPINLOCK(dev_list_lock);
-
 static DEFINE_IDA(nvme_instance_ida);
-
+static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
@@ -1031,20 +1025,12 @@ static int nvme_open(struct block_device *bdev, fmode_t mode)
 
 	if (!kref_get_unless_zero(&ns->kref))
 		return -ENXIO;
-	if (!try_module_get(ns->ctrl->ops->module)) {
-		kref_put(&ns->kref, nvme_free_ns);
-		return -ENXIO;
-	}
-
 	return 0;
 }
 
 static void nvme_release(struct gendisk *disk, fmode_t mode)
 {
-	struct nvme_ns *ns = disk->private_data;
-
-	module_put(ns->ctrl->ops->module);
-	nvme_put_ns(ns);
+	nvme_put_ns(disk->private_data);
 }
 
 static int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo)
@@ -1902,33 +1888,12 @@ EXPORT_SYMBOL_GPL(nvme_init_identify);
 
 static int nvme_dev_open(struct inode *inode, struct file *file)
 {
-	struct nvme_ctrl *ctrl;
-	int instance = iminor(inode);
-	int ret = -ENODEV;
-
-	spin_lock(&dev_list_lock);
-	list_for_each_entry(ctrl, &nvme_ctrl_list, node) {
-		if (ctrl->instance != instance)
-			continue;
-
-		if (!ctrl->admin_q) {
-			ret = -EWOULDBLOCK;
-			break;
-		}
-		if (!kobject_get_unless_zero(&ctrl->device->kobj))
-			break;
-		file->private_data = ctrl;
-		ret = 0;
-		break;
-	}
-	spin_unlock(&dev_list_lock);
-
-	return ret;
-}
+	struct nvme_ctrl *ctrl =
+		container_of(inode->i_cdev, struct nvme_ctrl, cdev);
 
-static int nvme_dev_release(struct inode *inode, struct file *file)
-{
-	nvme_put_ctrl(file->private_data);
+	if (!ctrl->admin_q)
+		return -EWOULDBLOCK;
+	file->private_data = ctrl;
 	return 0;
 }
 
@@ -1992,7 +1957,6 @@ static long nvme_dev_ioctl(struct file *file, unsigned int cmd,
 static const struct file_operations nvme_dev_fops = {
 	.owner		= THIS_MODULE,
 	.open		= nvme_dev_open,
-	.release	= nvme_dev_release,
 	.unlocked_ioctl	= nvme_dev_ioctl,
 	.compat_ioctl	= nvme_dev_ioctl,
 };
@@ -2703,11 +2667,7 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
-	device_del(ctrl->device);
-
-	spin_lock(&dev_list_lock);
-	list_del(&ctrl->node);
-	spin_unlock(&dev_list_lock);
+	cdev_device_del(&ctrl->cdev, ctrl->device);
 }
 EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
 
@@ -2750,7 +2710,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	device_initialize(&ctrl->ctrl_device);
 	ctrl->device = &ctrl->ctrl_device;
-	ctrl->device->devt = MKDEV(nvme_char_major, ctrl->instance);
+	ctrl->device->devt = MKDEV(MAJOR(nvme_chr_devt), ctrl->instance);
 	ctrl->device->class = nvme_class;
 	ctrl->device->parent = ctrl->dev;
 	ctrl->device->groups = nvme_dev_attr_groups;
@@ -2759,16 +2719,15 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	ret = dev_set_name(ctrl->device, "nvme%d", ctrl->instance);
 	if (ret)
 		goto out_release_instance;
-	ret = device_add(ctrl->device);
+
+	cdev_init(&ctrl->cdev, &nvme_dev_fops);
+	ctrl->cdev.owner = ops->module;
+	ret = cdev_device_add(&ctrl->cdev, ctrl->device);
 	if (ret)
 		goto out_free_name;
 
 	ida_init(&ctrl->ns_ida);
 
-	spin_lock(&dev_list_lock);
-	list_add_tail(&ctrl->node, &nvme_ctrl_list);
-	spin_unlock(&dev_list_lock);
-
 	/*
 	 * Initialize latency tolerance controls.  The sysfs files won't
 	 * be visible to userspace unless the device actually supports APST.
@@ -2899,12 +2858,9 @@ int __init nvme_core_init(void)
 	if (!nvme_wq)
 		return -ENOMEM;
 
-	result = __register_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme",
-							&nvme_dev_fops);
+	result = alloc_chrdev_region(&nvme_chr_devt, 0, NVME_MINORS, "nvme");
 	if (result < 0)
 		goto destroy_wq;
-	else if (result > 0)
-		nvme_char_major = result;
 
 	nvme_class = class_create(THIS_MODULE, "nvme");
 	if (IS_ERR(nvme_class)) {
@@ -2915,7 +2871,7 @@ int __init nvme_core_init(void)
 	return 0;
 
 unregister_chrdev:
-	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 destroy_wq:
 	destroy_workqueue(nvme_wq);
 	return result;
@@ -2924,7 +2880,7 @@ int __init nvme_core_init(void)
 void nvme_core_exit(void)
 {
 	class_destroy(nvme_class);
-	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 	destroy_workqueue(nvme_wq);
 }
 
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 55286acd5387..e16346048b73 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -15,6 +15,7 @@
 #define _NVME_H
 
 #include <linux/nvme.h>
+#include <linux/cdev.h>
 #include <linux/pci.h>
 #include <linux/kref.h>
 #include <linux/blk-mq.h>
@@ -134,7 +135,7 @@ struct nvme_ctrl {
 	struct mutex namespaces_mutex;
 	struct device ctrl_device;
 	struct device *device;	/* char device */
-	struct list_head node;
+	struct cdev cdev;
 	struct ida ns_ida;
 	struct work_struct reset_work;
 
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:53 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:53 +0200
Subject: [PATCH 12/17] nvme: check for a live controller in nvme_dev_open
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-13-hch@lst.de>

This is a much more sensible check than just the admin queue.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3757f5695de2..d490abd0195c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1891,7 +1891,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file)
 	struct nvme_ctrl *ctrl =
 		container_of(inode->i_cdev, struct nvme_ctrl, cdev);
 
-	if (!ctrl->admin_q)
+	if (ctrl->state != NVME_CTRL_LIVE)
 		return -EWOULDBLOCK;
 	file->private_data = ctrl;
 	return 0;
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:54 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:54 +0200
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-14-hch@lst.de>

This adds a new nvme_subsystem structure so that we can track multiple
controllers that belong to a single subsystem.  For now we only use it
to store the NQN, and to check that we don't have duplicate NQNs unless
the involved subsystems support multiple controllers.

Includes code originally from Hannes Reinecke to expose the subsystems
in sysfs.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c    | 179 +++++++++++++++++++++++++++++++++++++-------
 drivers/nvme/host/fabrics.c |   4 +-
 drivers/nvme/host/nvme.h    |  20 +++--
 3 files changed, 167 insertions(+), 36 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d490abd0195c..88f67deabf7f 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -68,9 +68,13 @@ MODULE_PARM_DESC(streams, "turn on support for Streams write directives");
 struct workqueue_struct *nvme_wq;
 EXPORT_SYMBOL_GPL(nvme_wq);
 
+static LIST_HEAD(nvme_subsystems);
+static DEFINE_MUTEX(nvme_subsystems_lock);
+
 static DEFINE_IDA(nvme_instance_ida);
 static dev_t nvme_chr_devt;
 static struct class *nvme_class;
+static struct class *nvme_subsys_class;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
 {
@@ -1694,14 +1698,15 @@ static bool quirk_matches(const struct nvme_id_ctrl *id,
 		string_matches(id->fr, q->fr, sizeof(id->fr));
 }
 
-static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+static void nvme_init_subnqn(struct nvme_subsystem *subsys, struct nvme_ctrl *ctrl,
+		struct nvme_id_ctrl *id)
 {
 	size_t nqnlen;
 	int off;
 
 	nqnlen = strnlen(id->subnqn, NVMF_NQN_SIZE);
 	if (nqnlen > 0 && nqnlen < NVMF_NQN_SIZE) {
-		strcpy(ctrl->subnqn, id->subnqn);
+		strcpy(subsys->subnqn, id->subnqn);
 		return;
 	}
 
@@ -1709,14 +1714,112 @@ static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 		dev_warn(ctrl->device, "missing or invalid SUBNQN field.\n");
 
 	/* Generate a "fake" NQN per Figure 254 in NVMe 1.3 + ECN 001 */
-	off = snprintf(ctrl->subnqn, NVMF_NQN_SIZE,
+	off = snprintf(subsys->subnqn, NVMF_NQN_SIZE,
 			"nqn.2014.08.org.nvmexpress:%4x%4x",
 			le16_to_cpu(id->vid), le16_to_cpu(id->ssvid));
-	memcpy(ctrl->subnqn + off, id->sn, sizeof(id->sn));
+	memcpy(subsys->subnqn + off, id->sn, sizeof(id->sn));
 	off += sizeof(id->sn);
-	memcpy(ctrl->subnqn + off, id->mn, sizeof(id->mn));
+	memcpy(subsys->subnqn + off, id->mn, sizeof(id->mn));
 	off += sizeof(id->mn);
-	memset(ctrl->subnqn + off, 0, sizeof(ctrl->subnqn) - off);
+	memset(subsys->subnqn + off, 0, sizeof(subsys->subnqn) - off);
+}
+
+static void nvme_free_subsystem(struct device *dev)
+{
+	struct nvme_subsystem *subsys =
+			container_of(dev, struct nvme_subsystem, dev);
+
+	mutex_lock(&nvme_subsystems_lock);
+	list_del(&subsys->entry);
+	mutex_unlock(&nvme_subsystems_lock);
+
+	kfree(subsys);
+}
+
+static void nvme_put_subsystem(struct nvme_subsystem *subsys)
+{
+	put_device(&subsys->dev);
+}
+
+static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
+{
+	struct nvme_subsystem *subsys;
+
+	lockdep_assert_held(&nvme_subsystems_lock);
+
+	list_for_each_entry(subsys, &nvme_subsystems, entry) {
+		if (strcmp(subsys->subnqn, subsysnqn))
+			continue;
+		if (!kobject_get_unless_zero(&subsys->dev.kobj))
+			continue;
+		return subsys;
+	}
+
+	return NULL;
+}
+
+static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+{
+	struct nvme_subsystem *subsys, *found;
+	int ret;
+
+	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
+	if (!subsys)
+		return -ENOMEM;
+	INIT_LIST_HEAD(&subsys->ctrls);
+	nvme_init_subnqn(subsys, ctrl, id);
+	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
+	memcpy(subsys->model, id->mn, sizeof(subsys->model));
+	memcpy(subsys->firmware_rev, id->fr, sizeof(subsys->firmware_rev));
+	subsys->vendor_id = le16_to_cpu(id->vid);
+	mutex_init(&subsys->lock);
+
+	subsys->dev.class = nvme_subsys_class;
+	subsys->dev.release = nvme_free_subsystem;
+	dev_set_name(&subsys->dev, subsys->subnqn);
+	device_initialize(&subsys->dev);
+
+	mutex_lock(&nvme_subsystems_lock);
+	found = __nvme_find_get_subsystem(subsys->subnqn);
+	if (found) {
+		/*
+		 * Verify that the subsystem actually supports multiple
+		 * controllers, else bail out.
+		 */
+		if (!(id->cmic & (1 << 1))) {
+			dev_err(ctrl->device,
+				"ignoring ctrl due to duplicate subnqn (%s).\n",
+				found->subnqn);
+			nvme_put_subsystem(found);
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		kfree(subsys);
+		subsys = found;
+	} else {
+		ret = device_add(&subsys->dev);
+		if (ret) {
+			dev_err(ctrl->device,
+				"failed to register subsystem device.\n");
+			goto out_unlock;
+		}
+		list_add_tail(&subsys->entry, &nvme_subsystems);
+	}
+
+	ctrl->subsys = subsys;
+	mutex_unlock(&nvme_subsystems_lock);
+
+	mutex_lock(&subsys->lock);
+	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
+	mutex_unlock(&subsys->lock);
+
+	return 0;
+
+out_unlock:
+	mutex_unlock(&nvme_subsystems_lock);
+	kfree(subsys);
+	return ret;
 }
 
 /*
@@ -1754,9 +1857,13 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 		return -EIO;
 	}
 
-	nvme_init_subnqn(ctrl, id);
-
 	if (!ctrl->identified) {
+		int i;
+
+		ret = nvme_init_subsystem(ctrl, id);
+		if (ret)
+			goto out_free;
+
 		/*
 		 * Check for quirks.  Quirk can depend on firmware version,
 		 * so, in principle, the set of quirks present can change
@@ -1765,9 +1872,6 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 		 * the device, but we'd have to make sure that the driver
 		 * behaves intelligently if the quirks change.
 		 */
-
-		int i;
-
 		for (i = 0; i < ARRAY_SIZE(core_quirks); i++) {
 			if (quirk_matches(id, &core_quirks[i]))
 				ctrl->quirks |= core_quirks[i].quirks;
@@ -1780,14 +1884,10 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 	}
 
 	ctrl->oacs = le16_to_cpu(id->oacs);
-	ctrl->vid = le16_to_cpu(id->vid);
 	ctrl->oncs = le16_to_cpup(&id->oncs);
 	atomic_set(&ctrl->abort_limit, id->acl + 1);
 	ctrl->vwc = id->vwc;
 	ctrl->cntlid = le16_to_cpup(&id->cntlid);
-	memcpy(ctrl->serial, id->sn, sizeof(id->sn));
-	memcpy(ctrl->model, id->mn, sizeof(id->mn));
-	memcpy(ctrl->firmware_rev, id->fr, sizeof(id->fr));
 	if (id->mdts)
 		max_hw_sectors = 1 << (id->mdts + page_shift - 9);
 	else
@@ -1990,9 +2090,9 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ctrl *ctrl = ns->ctrl;
-	int serial_len = sizeof(ctrl->serial);
-	int model_len = sizeof(ctrl->model);
+	struct nvme_subsystem *subsys = ns->ctrl->subsys;
+	int serial_len = sizeof(subsys->serial);
+	int model_len = sizeof(subsys->model);
 
 	if (!uuid_is_null(&ns->uuid))
 		return sprintf(buf, "uuid.%pU\n", &ns->uuid);
@@ -2003,15 +2103,16 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 	if (memchr_inv(ns->eui, 0, sizeof(ns->eui)))
 		return sprintf(buf, "eui.%8phN\n", ns->eui);
 
-	while (serial_len > 0 && (ctrl->serial[serial_len - 1] == ' ' ||
-				  ctrl->serial[serial_len - 1] == '\0'))
+	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
+				  subsys->serial[serial_len - 1] == '\0'))
 		serial_len--;
-	while (model_len > 0 && (ctrl->model[model_len - 1] == ' ' ||
-				 ctrl->model[model_len - 1] == '\0'))
+	while (model_len > 0 && (subsys->model[model_len - 1] == ' ' ||
+				 subsys->model[model_len - 1] == '\0'))
 		model_len--;
 
-	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", ctrl->vid,
-		serial_len, ctrl->serial, model_len, ctrl->model, ns->ns_id);
+	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
+		serial_len, subsys->serial, model_len, subsys->model,
+		ns->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
@@ -2097,10 +2198,15 @@ static ssize_t  field##_show(struct device *dev,				\
 			    struct device_attribute *attr, char *buf)		\
 {										\
         struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
-        return sprintf(buf, "%.*s\n", (int)sizeof(ctrl->field), ctrl->field);	\
+        return sprintf(buf, "%.*s\n",						\
+		(int)sizeof(ctrl->subsys->field), ctrl->subsys->field);		\
 }										\
 static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
 
+nvme_show_str_function(model);
+nvme_show_str_function(serial);
+nvme_show_str_function(firmware_rev);
+
 #define nvme_show_int_function(field)						\
 static ssize_t  field##_show(struct device *dev,				\
 			    struct device_attribute *attr, char *buf)		\
@@ -2110,9 +2216,6 @@ static ssize_t  field##_show(struct device *dev,				\
 }										\
 static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
 
-nvme_show_str_function(model);
-nvme_show_str_function(serial);
-nvme_show_str_function(firmware_rev);
 nvme_show_int_function(cntlid);
 
 static ssize_t nvme_sysfs_delete(struct device *dev,
@@ -2166,7 +2269,7 @@ static ssize_t nvme_sysfs_show_subsysnqn(struct device *dev,
 {
 	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
 
-	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subnqn);
+	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subsys->subnqn);
 }
 static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
 
@@ -2675,11 +2778,21 @@ static void nvme_free_ctrl(struct device *dev)
 {
 	struct nvme_ctrl *ctrl =
 		container_of(dev, struct nvme_ctrl, ctrl_device);
+	struct nvme_subsystem *subsys = ctrl->subsys;
 
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 	ida_destroy(&ctrl->ns_ida);
 
+	if (subsys) {
+		mutex_lock(&subsys->lock);
+		list_del(&ctrl->subsys_entry);
+		mutex_unlock(&subsys->lock);
+	}
+
 	ctrl->ops->free_ctrl(ctrl);
+
+	if (subsys)
+		nvme_put_subsystem(subsys);
 }
 
 /*
@@ -2868,8 +2981,15 @@ int __init nvme_core_init(void)
 		goto unregister_chrdev;
 	}
 
+	nvme_subsys_class = class_create(THIS_MODULE, "nvme-subsystem");
+	if (IS_ERR(nvme_subsys_class)) {
+		result = PTR_ERR(nvme_subsys_class);
+		goto destroy_class;
+	}
 	return 0;
 
+destroy_class:
+	class_destroy(nvme_class);
 unregister_chrdev:
 	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 destroy_wq:
@@ -2879,6 +2999,7 @@ int __init nvme_core_init(void)
 
 void nvme_core_exit(void)
 {
+	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
 	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 	destroy_workqueue(nvme_wq);
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 8bca36a46924..dcad66e37a8f 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -877,10 +877,10 @@ nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
 		goto out_unlock;
 	}
 
-	if (strcmp(ctrl->subnqn, opts->subsysnqn)) {
+	if (strcmp(ctrl->subsys->subnqn, opts->subsysnqn)) {
 		dev_warn(ctrl->device,
 			"controller returned incorrect NQN: \"%s\".\n",
-			ctrl->subnqn);
+			ctrl->subsys->subnqn);
 		up_read(&nvmf_transports_rwsem);
 		ctrl->ops->delete_ctrl(ctrl);
 		return ERR_PTR(-EINVAL);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index e16346048b73..c12705cf14a3 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -139,13 +139,12 @@ struct nvme_ctrl {
 	struct ida ns_ida;
 	struct work_struct reset_work;
 
+	struct nvme_subsystem *subsys;
+	struct list_head subsys_entry;
+
 	struct opal_dev *opal_dev;
 
 	char name[12];
-	char serial[20];
-	char model[40];
-	char firmware_rev[8];
-	char subnqn[NVMF_NQN_SIZE];
 	u16 cntlid;
 
 	u32 ctrl_config;
@@ -156,7 +155,6 @@ struct nvme_ctrl {
 	u32 page_size;
 	u32 max_hw_sectors;
 	u16 oncs;
-	u16 vid;
 	u16 oacs;
 	u16 nssa;
 	u16 nr_streams;
@@ -198,6 +196,18 @@ struct nvme_ctrl {
 	struct nvmf_ctrl_options *opts;
 };
 
+struct nvme_subsystem {
+	struct device		dev;
+	struct list_head	entry;
+	struct mutex		lock;
+	struct list_head	ctrls;
+	char			subnqn[NVMF_NQN_SIZE];
+	char			serial[20];
+	char			model[40];
+	char			firmware_rev[8];
+	u16			vendor_id;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:55 +0200
Subject: [PATCH 14/17] nvme: introduce a nvme_ns_ids structure
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-15-hch@lst.de>

This allows us to manage the various uniqueue namespace identifiers
together instead needing various variables and arguments.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 69 +++++++++++++++++++++++++++---------------------
 drivers/nvme/host/nvme.h | 14 +++++++---
 2 files changed, 49 insertions(+), 34 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 88f67deabf7f..30a60643881b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -749,7 +749,7 @@ static int nvme_identify_ctrl(struct nvme_ctrl *dev, struct nvme_id_ctrl **id)
 }
 
 static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
-		u8 *eui64, u8 *nguid, uuid_t *uuid)
+		struct nvme_ns_ids *ids)
 {
 	struct nvme_command c = { };
 	int status;
@@ -785,7 +785,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_EUI64_LEN;
-			memcpy(eui64, data + pos + sizeof(*cur), len);
+			memcpy(ids->eui64, data + pos + sizeof(*cur), len);
 			break;
 		case NVME_NIDT_NGUID:
 			if (cur->nidl != NVME_NIDT_NGUID_LEN) {
@@ -795,7 +795,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_NGUID_LEN;
-			memcpy(nguid, data + pos + sizeof(*cur), len);
+			memcpy(ids->nguid, data + pos + sizeof(*cur), len);
 			break;
 		case NVME_NIDT_UUID:
 			if (cur->nidl != NVME_NIDT_UUID_LEN) {
@@ -805,7 +805,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_UUID_LEN;
-			uuid_copy(uuid, data + pos + sizeof(*cur));
+			uuid_copy(&ids->uuid, data + pos + sizeof(*cur));
 			break;
 		default:
 			/* Skip unnkown types */
@@ -1137,22 +1137,31 @@ static void nvme_config_discard(struct nvme_ns *ns)
 }
 
 static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
-		struct nvme_id_ns *id, u8 *eui64, u8 *nguid, uuid_t *uuid)
+		struct nvme_id_ns *id, struct nvme_ns_ids *ids)
 {
+	memset(ids, 0, sizeof(*ids));
+
 	if (ctrl->vs >= NVME_VS(1, 1, 0))
-		memcpy(eui64, id->eui64, sizeof(id->eui64));
+		memcpy(ids->eui64, id->eui64, sizeof(id->eui64));
 	if (ctrl->vs >= NVME_VS(1, 2, 0))
-		memcpy(nguid, id->nguid, sizeof(id->nguid));
+		memcpy(ids->nguid, id->nguid, sizeof(id->nguid));
 	if (ctrl->vs >= NVME_VS(1, 3, 0)) {
 		 /* Don't treat error as fatal we potentially
 		  * already have a NGUID or EUI-64
 		  */
-		if (nvme_identify_ns_descs(ctrl, nsid, eui64, nguid, uuid))
+		if (nvme_identify_ns_descs(ctrl, nsid, ids))
 			dev_warn(ctrl->device,
 				 "%s: Identify Descriptors failed\n", __func__);
 	}
 }
 
+static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b)
+{
+	return uuid_equal(&a->uuid, &b->uuid) &&
+		memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) == 0 &&
+		memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) == 0;
+}
+
 static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
 {
 	struct nvme_ns *ns = disk->private_data;
@@ -1193,8 +1202,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 	struct nvme_ns *ns = disk->private_data;
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	struct nvme_id_ns *id;
-	u8 eui64[8] = { 0 }, nguid[16] = { 0 };
-	uuid_t uuid = uuid_null;
+	struct nvme_ns_ids ids;
 	int ret = 0;
 
 	if (test_bit(NVME_NS_DEAD, &ns->flags)) {
@@ -1211,10 +1219,8 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		goto out;
 	}
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, eui64, nguid, &uuid);
-	if (!uuid_equal(&ns->uuid, &uuid) ||
-	    memcmp(&ns->nguid, &nguid, sizeof(ns->nguid)) ||
-	    memcmp(&ns->eui, &eui64, sizeof(ns->eui))) {
+	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ids);
+	if (!nvme_ns_ids_equal(&ns->ids, &ids)) {
 		dev_err(ctrl->device,
 			"identifiers changed for nsid %d\n", ns->ns_id);
 		ret = -ENODEV;
@@ -2090,18 +2096,19 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 	struct nvme_subsystem *subsys = ns->ctrl->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
 
-	if (!uuid_is_null(&ns->uuid))
-		return sprintf(buf, "uuid.%pU\n", &ns->uuid);
+	if (!uuid_is_null(&ids->uuid))
+		return sprintf(buf, "uuid.%pU\n", &ids->uuid);
 
-	if (memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
-		return sprintf(buf, "eui.%16phN\n", ns->nguid);
+	if (memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
+		return sprintf(buf, "eui.%16phN\n", ids->nguid);
 
-	if (memchr_inv(ns->eui, 0, sizeof(ns->eui)))
-		return sprintf(buf, "eui.%8phN\n", ns->eui);
+	if (memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
+		return sprintf(buf, "eui.%8phN\n", ids->eui64);
 
 	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
 				  subsys->serial[serial_len - 1] == '\0'))
@@ -2120,7 +2127,7 @@ static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
 			  char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->nguid);
+	return sprintf(buf, "%pU\n", ns->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
@@ -2128,16 +2135,17 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
 	 */
-	if (uuid_is_null(&ns->uuid)) {
+	if (uuid_is_null(&ids->uuid)) {
 		printk_ratelimited(KERN_WARNING
 				   "No UUID available providing old NGUID\n");
-		return sprintf(buf, "%pU\n", ns->nguid);
+		return sprintf(buf, "%pU\n", ids->nguid);
 	}
-	return sprintf(buf, "%pU\n", &ns->uuid);
+	return sprintf(buf, "%pU\n", &ids->uuid);
 }
 static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
 
@@ -2145,7 +2153,7 @@ static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->eui);
+	return sprintf(buf, "%8phd\n", ns->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
@@ -2171,18 +2179,19 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 
 	if (a == &dev_attr_uuid.attr) {
-		if (uuid_is_null(&ns->uuid) ||
-		    !memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+		if (uuid_is_null(&ids->uuid) ||
+		    !memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
 			return 0;
 	}
 	if (a == &dev_attr_nguid.attr) {
-		if (!memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+		if (!memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
 			return 0;
 	}
 	if (a == &dev_attr_eui.attr) {
-		if (!memchr_inv(ns->eui, 0, sizeof(ns->eui)))
+		if (!memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
 			return 0;
 	}
 	return a->mode;
@@ -2415,7 +2424,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, ns->eui, ns->nguid, &ns->uuid);
+	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ns->ids);
 
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index c12705cf14a3..650e38d21fd2 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -208,6 +208,15 @@ struct nvme_subsystem {
 	u16			vendor_id;
 };
 
+/*
+ * Container structure for uniqueue namespace identifiers.
+ */
+struct nvme_ns_ids {
+	u8	eui64[8];
+	u8	nguid[16];
+	uuid_t	uuid;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
@@ -218,11 +227,8 @@ struct nvme_ns {
 	struct kref kref;
 	int instance;
 
-	u8 eui[8];
-	u8 nguid[16];
-	uuid_t uuid;
-
 	unsigned ns_id;
+	struct nvme_ns_ids ids;
 	int lba_shift;
 	u16 ms;
 	u16 sgs;
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:56 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:56 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-16-hch@lst.de>

Introduce a new struct nvme_ns_head that holds information about an actual
namespace, unlike struct nvme_ns, which only holds the per-controller
namespace information.  For private namespaces there is a 1:1 relation of
the two, but for shared namespaces this lets us discover all the paths to
it.  For now only the identifiers are moved to the new structure, but most
of the information in struct nvme_ns should eventually move over.

To allow lockless path lookup the list of nvme_ns structures per
nvme_ns_head is protected by SRCU, which requires freeing the nvme_ns
structure through call_srcu.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c     | 192 +++++++++++++++++++++++++++++++++++++------
 drivers/nvme/host/lightnvm.c |  14 ++--
 drivers/nvme/host/nvme.h     |  21 ++++-
 3 files changed, 192 insertions(+), 35 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 30a60643881b..d294792d8584 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -244,10 +244,28 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 }
 EXPORT_SYMBOL_GPL(nvme_change_ctrl_state);
 
+static void nvme_free_ns_head(struct kref *ref)
+{
+	struct nvme_ns_head *head =
+		container_of(ref, struct nvme_ns_head, ref);
+
+	list_del_init(&head->entry);
+	cleanup_srcu_struct(&head->srcu);
+	kfree(head);
+}
+
+static void nvme_put_ns_head(struct nvme_ns_head *head)
+{
+	kref_put(&head->ref, nvme_free_ns_head);
+}
+
 static void nvme_free_ns(struct kref *kref)
 {
 	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
 
+	if (ns->head)
+		nvme_put_ns_head(ns->head);
+
 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
 
@@ -388,7 +406,7 @@ static inline void nvme_setup_flush(struct nvme_ns *ns,
 {
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->common.opcode = nvme_cmd_flush;
-	cmnd->common.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->common.nsid = cpu_to_le32(ns->head->ns_id);
 }
 
 static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req,
@@ -419,7 +437,7 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req,
 
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->dsm.opcode = nvme_cmd_dsm;
-	cmnd->dsm.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->dsm.nsid = cpu_to_le32(ns->head->ns_id);
 	cmnd->dsm.nr = cpu_to_le32(segments - 1);
 	cmnd->dsm.attributes = cpu_to_le32(NVME_DSMGMT_AD);
 
@@ -458,7 +476,7 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns,
 
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->rw.opcode = (rq_data_dir(req) ? nvme_cmd_write : nvme_cmd_read);
-	cmnd->rw.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->rw.nsid = cpu_to_le32(ns->head->ns_id);
 	cmnd->rw.slba = cpu_to_le64(nvme_block_nr(ns, blk_rq_pos(req)));
 	cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1);
 
@@ -939,7 +957,7 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 	memset(&c, 0, sizeof(c));
 	c.rw.opcode = io.opcode;
 	c.rw.flags = io.flags;
-	c.rw.nsid = cpu_to_le32(ns->ns_id);
+	c.rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c.rw.slba = cpu_to_le64(io.slba);
 	c.rw.length = cpu_to_le16(io.nblocks);
 	c.rw.control = cpu_to_le16(io.control);
@@ -1004,7 +1022,7 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 	switch (cmd) {
 	case NVME_IOCTL_ID:
 		force_successful_syscall_return();
-		return ns->ns_id;
+		return ns->head->ns_id;
 	case NVME_IOCTL_ADMIN_CMD:
 		return nvme_user_cmd(ns->ctrl, NULL, (void __user *)arg);
 	case NVME_IOCTL_IO_CMD:
@@ -1155,6 +1173,13 @@ static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
 	}
 }
 
+static bool nvme_ns_ids_valid(struct nvme_ns_ids *ids)
+{
+	return !uuid_is_null(&ids->uuid) ||
+		memchr_inv(ids->nguid, 0, sizeof(ids->nguid)) ||
+		memchr_inv(ids->eui64, 0, sizeof(ids->eui64));
+}
+
 static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b)
 {
 	return uuid_equal(&a->uuid, &b->uuid) &&
@@ -1210,7 +1235,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		return -ENODEV;
 	}
 
-	id = nvme_identify_ns(ctrl, ns->ns_id);
+	id = nvme_identify_ns(ctrl, ns->head->ns_id);
 	if (!id)
 		return -ENODEV;
 
@@ -1219,10 +1244,10 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		goto out;
 	}
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ids);
-	if (!nvme_ns_ids_equal(&ns->ids, &ids)) {
+	nvme_report_ns_ids(ctrl, ns->head->ns_id, id, &ids);
+	if (!nvme_ns_ids_equal(&ns->head->ids, &ids)) {
 		dev_err(ctrl->device,
-			"identifiers changed for nsid %d\n", ns->ns_id);
+			"identifiers changed for nsid %d\n", ns->head->ns_id);
 		ret = -ENODEV;
 	}
 
@@ -1263,7 +1288,7 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = op;
-	c.common.nsid = cpu_to_le32(ns->ns_id);
+	c.common.nsid = cpu_to_le32(ns->head->ns_id);
 	c.common.cdw10[0] = cpu_to_le32(cdw10);
 
 	return nvme_submit_sync_cmd(ns->queue, &c, data, 16);
@@ -1773,6 +1798,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	if (!subsys)
 		return -ENOMEM;
 	INIT_LIST_HEAD(&subsys->ctrls);
+	INIT_LIST_HEAD(&subsys->nsheads);
 	nvme_init_subnqn(subsys, ctrl, id);
 	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
 	memcpy(subsys->model, id->mn, sizeof(subsys->model));
@@ -2096,7 +2122,7 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 	struct nvme_subsystem *subsys = ns->ctrl->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
@@ -2119,7 +2145,7 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 
 	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
 		serial_len, subsys->serial, model_len, subsys->model,
-		ns->ns_id);
+		ns->head->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
@@ -2127,7 +2153,7 @@ static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
 			  char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->ids.nguid);
+	return sprintf(buf, "%pU\n", ns->head->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
@@ -2135,7 +2161,7 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
@@ -2153,7 +2179,7 @@ static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->ids.eui64);
+	return sprintf(buf, "%8phd\n", ns->head->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
@@ -2161,7 +2187,7 @@ static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%d\n", ns->ns_id);
+	return sprintf(buf, "%d\n", ns->head->ns_id);
 }
 static DEVICE_ATTR(nsid, S_IRUGO, nsid_show, NULL);
 
@@ -2179,7 +2205,7 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 
 	if (a == &dev_attr_uuid.attr) {
 		if (uuid_is_null(&ids->uuid) ||
@@ -2331,12 +2357,114 @@ static const struct attribute_group *nvme_dev_attr_groups[] = {
 	NULL,
 };
 
+static struct nvme_ns_head *__nvme_find_ns_head(struct nvme_subsystem *subsys,
+		unsigned nsid)
+{
+	struct nvme_ns_head *h;
+
+	lockdep_assert_held(&subsys->lock);
+
+	list_for_each_entry(h, &subsys->nsheads, entry) {
+		if (h->ns_id == nsid && kref_get_unless_zero(&h->ref))
+			return h;
+	}
+
+	return NULL;
+}
+
+static int __nvme_check_ids(struct nvme_subsystem *subsys,
+		struct nvme_ns_head *new)
+{
+	struct nvme_ns_head *h;
+
+	lockdep_assert_held(&subsys->lock);
+
+	list_for_each_entry(h, &subsys->nsheads, entry) {
+		if (nvme_ns_ids_valid(&new->ids) &&
+		    nvme_ns_ids_equal(&new->ids, &h->ids))
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
+		unsigned nsid, struct nvme_id_ns *id)
+{
+	struct nvme_ns_head *head;
+	int ret = -ENOMEM;
+
+	head = kzalloc(sizeof(*head), GFP_KERNEL);
+	if (!head)
+		goto out;
+
+	INIT_LIST_HEAD(&head->list);
+	head->ns_id = nsid;
+	init_srcu_struct(&head->srcu);
+	kref_init(&head->ref);
+
+	nvme_report_ns_ids(ctrl, nsid, id, &head->ids);
+
+	ret = __nvme_check_ids(ctrl->subsys, head);
+	if (ret) {
+		dev_err(ctrl->device,
+			"duplicate IDs for nsid %d\n", nsid);
+		goto out_free_head;
+	}
+
+	list_add_tail(&head->entry, &ctrl->subsys->nsheads);
+	return head;
+out_free_head:
+	cleanup_srcu_struct(&head->srcu);
+	kfree(head);
+out:
+	return ERR_PTR(ret);
+}
+
+static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
+		struct nvme_id_ns *id)
+{
+	struct nvme_ctrl *ctrl = ns->ctrl;
+	bool is_shared = id->nmic & (1 << 0);
+	struct nvme_ns_head *head = NULL;
+	int ret = 0;
+
+	mutex_lock(&ctrl->subsys->lock);
+	if (is_shared)
+		head = __nvme_find_ns_head(ctrl->subsys, nsid);
+	if (!head) {
+		head = nvme_alloc_ns_head(ctrl, nsid, id);
+		if (IS_ERR(head)) {
+			ret = PTR_ERR(head);
+			goto out_unlock;
+		}
+	} else {
+		struct nvme_ns_ids ids;
+
+		nvme_report_ns_ids(ctrl, nsid, id, &ids);
+		if (!nvme_ns_ids_equal(&head->ids, &ids)) {
+			dev_err(ctrl->device,
+				"IDs don't match for shared namespace %d\n",
+					nsid);
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+	}
+
+	list_add_tail(&ns->siblings, &head->list);
+	ns->head = head;
+
+out_unlock:
+	mutex_unlock(&ctrl->subsys->lock);
+	return ret;
+}
+
 static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 {
 	struct nvme_ns *nsa = container_of(a, struct nvme_ns, list);
 	struct nvme_ns *nsb = container_of(b, struct nvme_ns, list);
 
-	return nsa->ns_id - nsb->ns_id;
+	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
 static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
@@ -2345,13 +2473,13 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	mutex_lock(&ctrl->namespaces_mutex);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
-		if (ns->ns_id == nsid) {
+		if (ns->head->ns_id == nsid) {
 			if (!kref_get_unless_zero(&ns->kref))
 				continue;
 			ret = ns;
 			break;
 		}
-		if (ns->ns_id > nsid)
+		if (ns->head->ns_id > nsid)
 			break;
 	}
 	mutex_unlock(&ctrl->namespaces_mutex);
@@ -2366,7 +2494,7 @@ static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns)
 	if (!ctrl->nr_streams)
 		return 0;
 
-	ret = nvme_get_stream_params(ctrl, &s, ns->ns_id);
+	ret = nvme_get_stream_params(ctrl, &s, ns->head->ns_id);
 	if (ret)
 		return ret;
 
@@ -2408,7 +2536,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	ns->ctrl = ctrl;
 
 	kref_init(&ns->kref);
-	ns->ns_id = nsid;
 	ns->lba_shift = 9; /* set to a default value for 512 until disk is validated */
 
 	blk_queue_logical_block_size(ns->queue, 1 << ns->lba_shift);
@@ -2424,18 +2551,19 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ns->ids);
+	if (nvme_init_ns_head(ns, nsid, id))
+		goto out_free_id;
 
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
 			dev_warn(ctrl->device, "LightNVM init failure\n");
-			goto out_free_id;
+			goto out_unlink_ns;
 		}
 	}
 
 	disk = alloc_disk_node(0, node);
 	if (!disk)
-		goto out_free_id;
+		goto out_unlink_ns;
 
 	disk->fops = &nvme_fops;
 	disk->private_data = ns;
@@ -2463,6 +2591,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
 	return;
+ out_unlink_ns:
+	mutex_lock(&ctrl->subsys->lock);
+	list_del_rcu(&ns->siblings);
+	mutex_unlock(&ctrl->subsys->lock);
  out_free_id:
 	kfree(id);
  out_free_queue:
@@ -2475,6 +2607,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 static void nvme_ns_remove(struct nvme_ns *ns)
 {
+	struct nvme_ns_head *head = ns->head;
+
 	if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags))
 		return;
 
@@ -2489,10 +2623,16 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 		blk_cleanup_queue(ns->queue);
 	}
 
+	mutex_lock(&ns->ctrl->subsys->lock);
+	if (head)
+		list_del_rcu(&ns->siblings);
+	mutex_unlock(&ns->ctrl->subsys->lock);
+
 	mutex_lock(&ns->ctrl->namespaces_mutex);
 	list_del_init(&ns->list);
 	mutex_unlock(&ns->ctrl->namespaces_mutex);
 
+	synchronize_srcu(&head->srcu);
 	nvme_put_ns(ns);
 }
 
@@ -2515,7 +2655,7 @@ static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
 	struct nvme_ns *ns, *next;
 
 	list_for_each_entry_safe(ns, next, &ctrl->namespaces, list) {
-		if (ns->ns_id > nsid)
+		if (ns->head->ns_id > nsid)
 			nvme_ns_remove(ns);
 	}
 }
diff --git a/drivers/nvme/host/lightnvm.c b/drivers/nvme/host/lightnvm.c
index 1f79e3f141e6..44e46276319c 100644
--- a/drivers/nvme/host/lightnvm.c
+++ b/drivers/nvme/host/lightnvm.c
@@ -305,7 +305,7 @@ static int nvme_nvm_identity(struct nvm_dev *nvmdev, struct nvm_id *nvm_id)
 	int ret;
 
 	c.identity.opcode = nvme_nvm_admin_identity;
-	c.identity.nsid = cpu_to_le32(ns->ns_id);
+	c.identity.nsid = cpu_to_le32(ns->head->ns_id);
 	c.identity.chnl_off = 0;
 
 	nvme_nvm_id = kmalloc(sizeof(struct nvme_nvm_id), GFP_KERNEL);
@@ -344,7 +344,7 @@ static int nvme_nvm_get_l2p_tbl(struct nvm_dev *nvmdev, u64 slba, u32 nlb,
 	int ret = 0;
 
 	c.l2p.opcode = nvme_nvm_admin_get_l2p_tbl;
-	c.l2p.nsid = cpu_to_le32(ns->ns_id);
+	c.l2p.nsid = cpu_to_le32(ns->head->ns_id);
 	entries = kmalloc(len, GFP_KERNEL);
 	if (!entries)
 		return -ENOMEM;
@@ -402,7 +402,7 @@ static int nvme_nvm_get_bb_tbl(struct nvm_dev *nvmdev, struct ppa_addr ppa,
 	int ret = 0;
 
 	c.get_bb.opcode = nvme_nvm_admin_get_bb_tbl;
-	c.get_bb.nsid = cpu_to_le32(ns->ns_id);
+	c.get_bb.nsid = cpu_to_le32(ns->head->ns_id);
 	c.get_bb.spba = cpu_to_le64(ppa.ppa);
 
 	bb_tbl = kzalloc(tblsz, GFP_KERNEL);
@@ -452,7 +452,7 @@ static int nvme_nvm_set_bb_tbl(struct nvm_dev *nvmdev, struct ppa_addr *ppas,
 	int ret = 0;
 
 	c.set_bb.opcode = nvme_nvm_admin_set_bb_tbl;
-	c.set_bb.nsid = cpu_to_le32(ns->ns_id);
+	c.set_bb.nsid = cpu_to_le32(ns->head->ns_id);
 	c.set_bb.spba = cpu_to_le64(ppas->ppa);
 	c.set_bb.nlb = cpu_to_le16(nr_ppas - 1);
 	c.set_bb.value = type;
@@ -469,7 +469,7 @@ static inline void nvme_nvm_rqtocmd(struct nvm_rq *rqd, struct nvme_ns *ns,
 				    struct nvme_nvm_command *c)
 {
 	c->ph_rw.opcode = rqd->opcode;
-	c->ph_rw.nsid = cpu_to_le32(ns->ns_id);
+	c->ph_rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c->ph_rw.spba = cpu_to_le64(rqd->ppa_addr.ppa);
 	c->ph_rw.metadata = cpu_to_le64(rqd->dma_meta_list);
 	c->ph_rw.control = cpu_to_le16(rqd->flags);
@@ -691,7 +691,7 @@ static int nvme_nvm_submit_vio(struct nvme_ns *ns,
 
 	memset(&c, 0, sizeof(c));
 	c.ph_rw.opcode = vio.opcode;
-	c.ph_rw.nsid = cpu_to_le32(ns->ns_id);
+	c.ph_rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c.ph_rw.control = cpu_to_le16(vio.control);
 	c.ph_rw.length = cpu_to_le16(vio.nppas);
 
@@ -728,7 +728,7 @@ static int nvme_nvm_user_vcmd(struct nvme_ns *ns, int admin,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = vcmd.opcode;
-	c.common.nsid = cpu_to_le32(ns->ns_id);
+	c.common.nsid = cpu_to_le32(ns->head->ns_id);
 	c.common.cdw2[0] = cpu_to_le32(vcmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(vcmd.cdw3);
 	/* cdw11-12 */
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 650e38d21fd2..487641ab2ef2 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -201,6 +201,7 @@ struct nvme_subsystem {
 	struct list_head	entry;
 	struct mutex		lock;
 	struct list_head	ctrls;
+	struct list_head	nsheads;
 	char			subnqn[NVMF_NQN_SIZE];
 	char			serial[20];
 	char			model[40];
@@ -217,18 +218,34 @@ struct nvme_ns_ids {
 	uuid_t	uuid;
 };
 
+/*
+ * Anchor structure for namespaces.  There is one for each namespace in a
+ * NVMe subsystem that any of our controllers can see, and the namespace
+ * structure for each controller is chained of it.  For private namespaces
+ * there is a 1:1 relation to our namespace structures, that is ->list
+ * only ever has a single entry for private namespaces.
+ */
+struct nvme_ns_head {
+	struct list_head	list;
+	struct srcu_struct      srcu;
+	unsigned		ns_id;
+	struct nvme_ns_ids	ids;
+	struct list_head	entry;
+	struct kref		ref;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
 	struct nvme_ctrl *ctrl;
 	struct request_queue *queue;
 	struct gendisk *disk;
+	struct list_head siblings;
 	struct nvm_dev *ndev;
 	struct kref kref;
+	struct nvme_ns_head *head;
 	int instance;
 
-	unsigned ns_id;
-	struct nvme_ns_ids ids;
 	int lba_shift;
 	u16 ms;
 	u16 sgs;
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:57 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:57 +0200
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-17-hch@lst.de>

This patch adds native multipath support to the nvme driver.  For each
namespace we create only single block device node, which can be used
to access that namespace through any of the controllers that refer to it.
The gendisk for each controllers path to the name space still exists
inside the kernel, but is hidden from userspace.  The character device
nodes are still available on a per-controller basis.  A new link from
the sysfs directory for the subsystem allows to find all controllers
for a given subsystem.

Currently we will always send I/O to the first available path, this will
be changed once the NVMe Asynchronous Namespace Access (ANA) TP is
ratified and implemented, at which point we will look at the ANA state
for each namespace.  Another possibility that was prototyped is to
use the path that is closes to the submitting NUMA code, which will be
mostly interesting for PCI, but might also be useful for RDMA or FC
transports in the future.  There is not plan to implement round robin
or I/O service time path selectors, as those are not scalable with
the performance rates provided by NVMe.

The multipath device will go away once all paths to it disappear,
any delay to keep it alive needs to be implemented at the controller
level.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 369 ++++++++++++++++++++++++++++++++++++++++-------
 drivers/nvme/host/nvme.h |  16 +-
 2 files changed, 333 insertions(+), 52 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d294792d8584..49a81118c9f8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -69,6 +69,7 @@ struct workqueue_struct *nvme_wq;
 EXPORT_SYMBOL_GPL(nvme_wq);
 
 static LIST_HEAD(nvme_subsystems);
+static DEFINE_IDA(nvme_subsystems_ida);
 static DEFINE_MUTEX(nvme_subsystems_lock);
 
 static DEFINE_IDA(nvme_instance_ida);
@@ -101,6 +102,20 @@ static int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
 	return ret;
 }
 
+static void nvme_failover_req(struct request *req)
+{
+	struct nvme_ns *ns = req->q->queuedata;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ns->head->requeue_lock, flags);
+	blk_steal_bios(&ns->head->requeue_list, req);
+	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
+	blk_mq_end_request(req, 0);
+
+	nvme_reset_ctrl(ns->ctrl);
+	kblockd_schedule_work(&ns->head->requeue_work);
+}
+
 static blk_status_t nvme_error_status(struct request *req)
 {
 	switch (nvme_req(req)->status & 0x7ff) {
@@ -128,6 +143,53 @@ static blk_status_t nvme_error_status(struct request *req)
 	}
 }
 
+static bool nvme_req_needs_failover(struct request *req)
+{
+	if (!(req->cmd_flags & REQ_NVME_MPATH))
+		return false;
+
+	switch (nvme_req(req)->status & 0x7ff) {
+	/*
+	 * Generic command status:
+	 */
+	case NVME_SC_INVALID_OPCODE:
+	case NVME_SC_INVALID_FIELD:
+	case NVME_SC_INVALID_NS:
+	case NVME_SC_LBA_RANGE:
+	case NVME_SC_CAP_EXCEEDED:
+	case NVME_SC_RESERVATION_CONFLICT:
+		return false;
+
+	/*
+	 * I/O command set specific error.  Unfortunately these values are
+	 * reused for fabrics commands, but those should never get here.
+	 */
+	case NVME_SC_BAD_ATTRIBUTES:
+	case NVME_SC_INVALID_PI:
+	case NVME_SC_READ_ONLY:
+	case NVME_SC_ONCS_NOT_SUPPORTED:
+		WARN_ON_ONCE(nvme_req(req)->cmd->common.opcode ==
+			nvme_fabrics_command);
+		return false;
+
+	/*
+	 * Media and Data Integrity Errors:
+	 */
+	case NVME_SC_WRITE_FAULT:
+	case NVME_SC_READ_ERROR:
+	case NVME_SC_GUARD_CHECK:
+	case NVME_SC_APPTAG_CHECK:
+	case NVME_SC_REFTAG_CHECK:
+	case NVME_SC_COMPARE_FAILED:
+	case NVME_SC_ACCESS_DENIED:
+	case NVME_SC_UNWRITTEN_BLOCK:
+		return false;
+	}
+
+	/* Everything else could be a path failure, so should be retried */
+	return true;
+}
+
 static inline bool nvme_req_needs_retry(struct request *req)
 {
 	if (blk_noretry_request(req))
@@ -142,6 +204,11 @@ static inline bool nvme_req_needs_retry(struct request *req)
 void nvme_complete_rq(struct request *req)
 {
 	if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
+		if (nvme_req_needs_failover(req)) {
+			nvme_failover_req(req);
+			return;
+		}
+
 		nvme_req(req)->retries++;
 		blk_mq_requeue_request(req, true);
 		return;
@@ -170,6 +237,18 @@ void nvme_cancel_request(struct request *req, void *data, bool reserved)
 }
 EXPORT_SYMBOL_GPL(nvme_cancel_request);
 
+static void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+
+	mutex_lock(&ctrl->namespaces_mutex);
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (ns->head)
+			kblockd_schedule_work(&ns->head->requeue_work);
+	}
+	mutex_unlock(&ctrl->namespaces_mutex);
+}
+
 bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 		enum nvme_ctrl_state new_state)
 {
@@ -237,9 +316,10 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 
 	if (changed)
 		ctrl->state = new_state;
-
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 
+	if (changed && ctrl->state == NVME_CTRL_LIVE)
+		nvme_kick_requeue_lists(ctrl);
 	return changed;
 }
 EXPORT_SYMBOL_GPL(nvme_change_ctrl_state);
@@ -249,6 +329,15 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	del_gendisk(head->disk);
+	blk_set_queue_dying(head->disk->queue);
+	/* make sure all pending bios are cleaned up */
+	kblockd_schedule_work(&head->requeue_work);
+	flush_work(&head->requeue_work);
+	blk_cleanup_queue(head->disk->queue);
+	put_disk(head->disk);
+	ida_simple_remove(&head->subsys->ns_ida, head->instance);
+
 	list_del_init(&head->entry);
 	cleanup_srcu_struct(&head->srcu);
 	kfree(head);
@@ -263,14 +352,11 @@ static void nvme_free_ns(struct kref *kref)
 {
 	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
 
-	if (ns->head)
-		nvme_put_ns_head(ns->head);
-
 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
-
 	put_disk(ns->disk);
-	ida_simple_remove(&ns->ctrl->ns_ida, ns->instance);
+	if (ns->head)
+		nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
 	kfree(ns);
 }
@@ -1014,11 +1100,9 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	return status;
 }
 
-static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
-		unsigned int cmd, unsigned long arg)
+static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd,
+		unsigned long arg)
 {
-	struct nvme_ns *ns = bdev->bd_disk->private_data;
-
 	switch (cmd) {
 	case NVME_IOCTL_ID:
 		force_successful_syscall_return();
@@ -1082,8 +1166,10 @@ static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
 	if (blk_get_integrity(disk) &&
 	    (ns->pi_type != pi_type || ns->ms != old_ms ||
 	     bs != queue_logical_block_size(disk->queue) ||
-	     (ns->ms && ns->ext)))
+	     (ns->ms && ns->ext))) {
 		blk_integrity_unregister(disk);
+		blk_integrity_unregister(ns->head->disk);
+	}
 
 	ns->pi_type = pi_type;
 }
@@ -1111,7 +1197,9 @@ static void nvme_init_integrity(struct nvme_ns *ns)
 	}
 	integrity.tuple_size = ns->ms;
 	blk_integrity_register(ns->disk, &integrity);
+	blk_integrity_register(ns->head->disk, &integrity);
 	blk_queue_max_integrity_segments(ns->queue, 1);
+	blk_queue_max_integrity_segments(ns->head->disk->queue, 1);
 }
 #else
 static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
@@ -1129,7 +1217,7 @@ static void nvme_set_chunk_size(struct nvme_ns *ns)
 	blk_queue_chunk_sectors(ns->queue, rounddown_pow_of_two(chunk_size));
 }
 
-static void nvme_config_discard(struct nvme_ns *ns)
+static void nvme_config_discard(struct nvme_ns *ns, struct request_queue *queue)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	u32 logical_block_size = queue_logical_block_size(ns->queue);
@@ -1140,18 +1228,18 @@ static void nvme_config_discard(struct nvme_ns *ns)
 	if (ctrl->nr_streams && ns->sws && ns->sgs) {
 		unsigned int sz = logical_block_size * ns->sws * ns->sgs;
 
-		ns->queue->limits.discard_alignment = sz;
-		ns->queue->limits.discard_granularity = sz;
+		queue->limits.discard_alignment = sz;
+		queue->limits.discard_granularity = sz;
 	} else {
 		ns->queue->limits.discard_alignment = logical_block_size;
 		ns->queue->limits.discard_granularity = logical_block_size;
 	}
-	blk_queue_max_discard_sectors(ns->queue, UINT_MAX);
-	blk_queue_max_discard_segments(ns->queue, NVME_DSM_MAX_RANGES);
-	queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, ns->queue);
+	blk_queue_max_discard_sectors(queue, UINT_MAX);
+	blk_queue_max_discard_segments(queue, NVME_DSM_MAX_RANGES);
+	queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, queue);
 
 	if (ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES)
-		blk_queue_max_write_zeroes_sectors(ns->queue, UINT_MAX);
+		blk_queue_max_write_zeroes_sectors(queue, UINT_MAX);
 }
 
 static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
@@ -1208,17 +1296,25 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
 	if (ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)
 		nvme_prep_integrity(disk, id, bs);
 	blk_queue_logical_block_size(ns->queue, bs);
+	blk_queue_logical_block_size(ns->head->disk->queue, bs);
 	if (ns->noiob)
 		nvme_set_chunk_size(ns);
 	if (ns->ms && !blk_get_integrity(disk) && !ns->ext)
 		nvme_init_integrity(ns);
-	if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk))
+	if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk)) {
 		set_capacity(disk, 0);
-	else
+		if (ns->head)
+			set_capacity(ns->head->disk, 0);
+	} else {
 		set_capacity(disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
+		if (ns->head)
+			set_capacity(ns->head->disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
+	}
 
-	if (ctrl->oncs & NVME_CTRL_ONCS_DSM)
-		nvme_config_discard(ns);
+	if (ctrl->oncs & NVME_CTRL_ONCS_DSM) {
+		nvme_config_discard(ns, ns->queue);
+		nvme_config_discard(ns, ns->head->disk->queue);
+	}
 	blk_mq_unfreeze_queue(disk->queue);
 }
 
@@ -1256,6 +1352,29 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 	return ret;
 }
 
+static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head)
+{
+	struct nvme_ns *ns;
+
+	list_for_each_entry_rcu(ns, &head->list, siblings) {
+		if (ns->ctrl->state == NVME_CTRL_LIVE) {
+			rcu_assign_pointer(head->current_path, ns);
+			return ns;
+		}
+	}
+
+	return NULL;
+}
+
+static inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
+{
+	struct nvme_ns *ns = srcu_dereference(head->current_path, &head->srcu);
+
+	if (unlikely(!ns || ns->ctrl->state != NVME_CTRL_LIVE))
+		ns = __nvme_find_path(head);
+	return ns;
+}
+
 static char nvme_pr_type(enum pr_type type)
 {
 	switch (type) {
@@ -1279,8 +1398,10 @@ static char nvme_pr_type(enum pr_type type)
 static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 				u64 key, u64 sa_key, u8 op)
 {
-	struct nvme_ns *ns = bdev->bd_disk->private_data;
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+	struct nvme_ns *ns;
 	struct nvme_command c;
+	int srcu_idx, ret;
 	u8 data[16] = { 0, };
 
 	put_unaligned_le64(key, &data[0]);
@@ -1288,10 +1409,17 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = op;
-	c.common.nsid = cpu_to_le32(ns->head->ns_id);
+	c.common.nsid = cpu_to_le32(head->ns_id);
 	c.common.cdw10[0] = cpu_to_le32(cdw10);
 
-	return nvme_submit_sync_cmd(ns->queue, &c, data, 16);
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns))
+		ret = nvme_submit_sync_cmd(ns->queue, &c, data, 16);
+	else
+		ret = -EWOULDBLOCK;
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
 }
 
 static int nvme_pr_register(struct block_device *bdev, u64 old,
@@ -1370,15 +1498,17 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 EXPORT_SYMBOL_GPL(nvme_sec_submit);
 #endif /* CONFIG_BLK_SED_OPAL */
 
+/*
+ * While we don't expose the per-controller devices to userspace we still
+ * need valid file operations for them, for one because the block layer
+ * expects to use the owner field for module refcounting, and also because
+ * we call revalidate_disk internally.
+ */
 static const struct block_device_operations nvme_fops = {
 	.owner		= THIS_MODULE,
-	.ioctl		= nvme_ioctl,
-	.compat_ioctl	= nvme_ioctl,
 	.open		= nvme_open,
 	.release	= nvme_release,
-	.getgeo		= nvme_getgeo,
 	.revalidate_disk= nvme_revalidate_disk,
-	.pr_ops		= &nvme_pr_ops,
 };
 
 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u64 cap, bool enabled)
@@ -1764,6 +1894,8 @@ static void nvme_free_subsystem(struct device *dev)
 	list_del(&subsys->entry);
 	mutex_unlock(&nvme_subsystems_lock);
 
+	ida_destroy(&subsys->ns_ida);
+	ida_simple_remove(&nvme_subsystems_ida, subsys->instance);
 	kfree(subsys);
 }
 
@@ -1792,11 +1924,15 @@ static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
 static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 {
 	struct nvme_subsystem *subsys, *found;
-	int ret;
+	int ret = -ENOMEM;
 
 	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
 	if (!subsys)
 		return -ENOMEM;
+	ret = ida_simple_get(&nvme_subsystems_ida, 0, 0, GFP_KERNEL);
+	if (ret < 0)
+		goto out_free_subsys;
+	subsys->instance = ret;
 	INIT_LIST_HEAD(&subsys->ctrls);
 	INIT_LIST_HEAD(&subsys->nsheads);
 	nvme_init_subnqn(subsys, ctrl, id);
@@ -1805,6 +1941,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	memcpy(subsys->firmware_rev, id->fr, sizeof(subsys->firmware_rev));
 	subsys->vendor_id = le16_to_cpu(id->vid);
 	mutex_init(&subsys->lock);
+	ida_init(&subsys->ns_ida);
 
 	subsys->dev.class = nvme_subsys_class;
 	subsys->dev.release = nvme_free_subsystem;
@@ -1827,6 +1964,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 			goto out_unlock;
 		}
 
+		ida_simple_remove(&nvme_subsystems_ida, subsys->instance);
 		kfree(subsys);
 		subsys = found;
 	} else {
@@ -1842,6 +1980,14 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	ctrl->subsys = subsys;
 	mutex_unlock(&nvme_subsystems_lock);
 
+	if (sysfs_create_link(&subsys->dev.kobj, &ctrl->device->kobj,
+			dev_name(ctrl->device))) {
+		dev_err(ctrl->device,
+			"failed to create sysfs link from subsystem.\n");
+		/* the transport driver will eventually put the subsystem */
+		return -EINVAL;
+	}
+
 	mutex_lock(&subsys->lock);
 	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
 	mutex_unlock(&subsys->lock);
@@ -1850,6 +1996,9 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 
 out_unlock:
 	mutex_unlock(&nvme_subsystems_lock);
+	ida_destroy(&subsys->ns_ida);
+	ida_simple_remove(&nvme_subsystems_ida, subsys->instance);
+out_free_subsys:
 	kfree(subsys);
 	return ret;
 }
@@ -2357,6 +2506,90 @@ static const struct attribute_group *nvme_dev_attr_groups[] = {
 	NULL,
 };
 
+static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,
+		struct bio *bio)
+{
+	struct nvme_ns_head *head = q->queuedata;
+	struct device *dev = disk_to_dev(head->disk);
+	struct nvme_ns *ns;
+	blk_qc_t ret = BLK_QC_T_NONE;
+	int srcu_idx;
+
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns)) {
+		bio->bi_disk = ns->disk;
+		bio->bi_opf |= REQ_NVME_MPATH;
+		ret = direct_make_request(bio);
+	} else if (!list_empty_careful(&head->list)) {
+		dev_warn_ratelimited(dev, "no path available - requeing I/O\n");
+
+		spin_lock_irq(&head->requeue_lock);
+		bio_list_add(&head->requeue_list, bio);
+		spin_unlock_irq(&head->requeue_lock);
+	} else {
+		dev_warn_ratelimited(dev, "no path - failing I/O\n");
+
+		bio->bi_status = BLK_STS_IOERR;
+		bio_endio(bio);
+	}
+
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
+}
+
+/*
+ * Issue the ioctl on the first available path.  Note that unlike normal block
+ * layer requests we will not retry failed request on another controller.
+ */
+static int nvme_ns_head_ioctl(struct block_device *bdev, fmode_t mode,
+		unsigned int cmd, unsigned long arg)
+{
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+	struct nvme_ns *ns;
+	int srcu_idx, ret;
+
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns))
+		ret = nvme_ns_ioctl(ns, cmd, arg);
+	else
+		ret = -EWOULDBLOCK;
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
+}
+
+static const struct block_device_operations nvme_ns_head_ops = {
+	.owner		= THIS_MODULE,
+	.ioctl		= nvme_ns_head_ioctl,
+	.compat_ioctl	= nvme_ns_head_ioctl,
+	.getgeo		= nvme_getgeo,
+	.pr_ops		= &nvme_pr_ops,
+};
+
+static void nvme_requeue_work(struct work_struct *work)
+{
+	struct nvme_ns_head *head =
+		container_of(work, struct nvme_ns_head, requeue_work);
+	struct bio *bio, *next;
+
+	spin_lock_irq(&head->requeue_lock);
+	next = bio_list_get(&head->requeue_list);
+	spin_unlock_irq(&head->requeue_lock);
+
+	while ((bio = next) != NULL) {
+		next = bio->bi_next;
+		bio->bi_next = NULL;
+
+		/*
+		 * Reset disk to the mpath node and resubmit to select a new
+		 * path.
+		 */
+		bio->bi_disk = head->disk;
+		generic_make_request(bio);
+	}
+}
+
 static struct nvme_ns_head *__nvme_find_ns_head(struct nvme_subsystem *subsys,
 		unsigned nsid)
 {
@@ -2392,15 +2625,23 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 		unsigned nsid, struct nvme_id_ns *id)
 {
 	struct nvme_ns_head *head;
+	struct request_queue *q;
 	int ret = -ENOMEM;
 
 	head = kzalloc(sizeof(*head), GFP_KERNEL);
 	if (!head)
 		goto out;
-
+	ret = ida_simple_get(&ctrl->subsys->ns_ida, 1, 0, GFP_KERNEL);
+	if (ret < 0)
+		goto out_free_head;
+	head->instance = ret;
 	INIT_LIST_HEAD(&head->list);
 	head->ns_id = nsid;
+	bio_list_init(&head->requeue_list);
+	spin_lock_init(&head->requeue_lock);
+	INIT_WORK(&head->requeue_work, nvme_requeue_work);
 	init_srcu_struct(&head->srcu);
+	head->subsys = ctrl->subsys;
 	kref_init(&head->ref);
 
 	nvme_report_ns_ids(ctrl, nsid, id, &head->ids);
@@ -2409,20 +2650,45 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 	if (ret) {
 		dev_err(ctrl->device,
 			"duplicate IDs for nsid %d\n", nsid);
-		goto out_free_head;
+		goto out_release_instance;
 	}
 
+	ret = -ENOMEM;
+	q = blk_alloc_queue_node(GFP_KERNEL, NUMA_NO_NODE);
+	if (!q)
+		goto out_free_head;
+	q->queuedata = head;
+	blk_queue_make_request(q, nvme_ns_head_make_request);
+	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
+	/* set to a default value for 512 until disk is validated */
+	blk_queue_logical_block_size(q, 512);
+	nvme_set_queue_limits(ctrl, q);
+
+	head->disk = alloc_disk(0);
+	if (!head->disk)
+		goto out_cleanup_queue;
+	head->disk->fops = &nvme_ns_head_ops;
+	head->disk->private_data = head;
+	head->disk->queue = q;
+	head->disk->flags = GENHD_FL_EXT_DEVT;
+	sprintf(head->disk->disk_name, "nvme%dn%d",
+			ctrl->subsys->instance, nsid);
 	list_add_tail(&head->entry, &ctrl->subsys->nsheads);
 	return head;
+
+out_cleanup_queue:
+	blk_cleanup_queue(q);
 out_free_head:
 	cleanup_srcu_struct(&head->srcu);
 	kfree(head);
+out_release_instance:
+	ida_simple_remove(&ctrl->subsys->ns_ida, head->instance);
 out:
 	return ERR_PTR(ret);
 }
 
 static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
-		struct nvme_id_ns *id)
+		struct nvme_id_ns *id, bool *new)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	bool is_shared = id->nmic & (1 << 0);
@@ -2438,6 +2704,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 			ret = PTR_ERR(head);
 			goto out_unlock;
 		}
+
+		*new = true;
 	} else {
 		struct nvme_ns_ids ids;
 
@@ -2449,6 +2717,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 			ret = -EINVAL;
 			goto out_unlock;
 		}
+
+		*new = false;
 	}
 
 	list_add_tail(&ns->siblings, &head->list);
@@ -2519,18 +2789,15 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	struct nvme_id_ns *id;
 	char disk_name[DISK_NAME_LEN];
 	int node = dev_to_node(ctrl->dev);
+	bool new = true;
 
 	ns = kzalloc_node(sizeof(*ns), GFP_KERNEL, node);
 	if (!ns)
 		return;
 
-	ns->instance = ida_simple_get(&ctrl->ns_ida, 1, 0, GFP_KERNEL);
-	if (ns->instance < 0)
-		goto out_free_ns;
-
 	ns->queue = blk_mq_init_queue(ctrl->tagset);
 	if (IS_ERR(ns->queue))
-		goto out_release_instance;
+		goto out_free_ns;
 	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, ns->queue);
 	ns->queue->queuedata = ns;
 	ns->ctrl = ctrl;
@@ -2542,8 +2809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	nvme_set_queue_limits(ctrl, ns->queue);
 	nvme_setup_streams_ns(ctrl, ns);
 
-	sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->instance);
-
 	id = nvme_identify_ns(ctrl, nsid);
 	if (!id)
 		goto out_free_queue;
@@ -2551,9 +2816,11 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	if (nvme_init_ns_head(ns, nsid, id))
+	if (nvme_init_ns_head(ns, nsid, id, &new))
 		goto out_free_id;
 
+	sprintf(disk_name, "nvme%dc%dn%d", ctrl->subsys->instance,
+			ctrl->cntlid, ns->head->instance);
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
 			dev_warn(ctrl->device, "LightNVM init failure\n");
@@ -2568,7 +2835,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	disk->fops = &nvme_fops;
 	disk->private_data = ns;
 	disk->queue = ns->queue;
-	disk->flags = GENHD_FL_EXT_DEVT;
+	disk->flags = GENHD_FL_HIDDEN;
 	memcpy(disk->disk_name, disk_name, DISK_NAME_LEN);
 	ns->disk = disk;
 
@@ -2590,6 +2857,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (ns->ndev && nvme_nvm_register_sysfs(ns))
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
+
+	if (new)
+		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
+
 	return;
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
@@ -2599,8 +2870,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	kfree(id);
  out_free_queue:
 	blk_cleanup_queue(ns->queue);
- out_release_instance:
-	ida_simple_remove(&ctrl->ns_ida, ns->instance);
  out_free_ns:
 	kfree(ns);
 }
@@ -2615,8 +2884,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		if (blk_get_integrity(ns->disk))
 			blk_integrity_unregister(ns->disk);
-		sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
-					&nvme_ns_attr_group);
 		if (ns->ndev)
 			nvme_nvm_unregister_sysfs(ns);
 		del_gendisk(ns->disk);
@@ -2624,8 +2891,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	}
 
 	mutex_lock(&ns->ctrl->subsys->lock);
-	if (head)
+	if (head) {
+		if (head->current_path == ns)
+			rcu_assign_pointer(head->current_path, NULL);
 		list_del_rcu(&ns->siblings);
+	}
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	mutex_lock(&ns->ctrl->namespaces_mutex);
@@ -2930,12 +3200,12 @@ static void nvme_free_ctrl(struct device *dev)
 	struct nvme_subsystem *subsys = ctrl->subsys;
 
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
-	ida_destroy(&ctrl->ns_ida);
 
 	if (subsys) {
 		mutex_lock(&subsys->lock);
 		list_del(&ctrl->subsys_entry);
 		mutex_unlock(&subsys->lock);
+		sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
 	}
 
 	ctrl->ops->free_ctrl(ctrl);
@@ -2988,8 +3258,6 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	if (ret)
 		goto out_free_name;
 
-	ida_init(&ctrl->ns_ida);
-
 	/*
 	 * Initialize latency tolerance controls.  The sysfs files won't
 	 * be visible to userspace unless the device actually supports APST.
@@ -3148,6 +3416,7 @@ int __init nvme_core_init(void)
 
 void nvme_core_exit(void)
 {
+	ida_destroy(&nvme_subsystems_ida);
 	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
 	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 487641ab2ef2..c84b6c2cfa6f 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -95,6 +95,11 @@ struct nvme_request {
 	u16			status;
 };
 
+/*
+ * Mark a bio as coming in through the mpath node.
+ */
+#define REQ_NVME_MPATH		REQ_DRV
+
 enum {
 	NVME_REQ_CANCELLED		= (1 << 0),
 };
@@ -136,7 +141,6 @@ struct nvme_ctrl {
 	struct device ctrl_device;
 	struct device *device;	/* char device */
 	struct cdev cdev;
-	struct ida ns_ida;
 	struct work_struct reset_work;
 
 	struct nvme_subsystem *subsys;
@@ -207,6 +211,8 @@ struct nvme_subsystem {
 	char			model[40];
 	char			firmware_rev[8];
 	u16			vendor_id;
+	int			instance;
+	struct ida		ns_ida;
 };
 
 /*
@@ -226,12 +232,19 @@ struct nvme_ns_ids {
  * only ever has a single entry for private namespaces.
  */
 struct nvme_ns_head {
+	struct nvme_ns __rcu	*current_path;
+	struct gendisk		*disk;
 	struct list_head	list;
 	struct srcu_struct      srcu;
+	struct nvme_subsystem	*subsys;
+	struct bio_list		requeue_list;
+	spinlock_t		requeue_lock;
+	struct work_struct	requeue_work;
 	unsigned		ns_id;
 	struct nvme_ns_ids	ids;
 	struct list_head	entry;
 	struct kref		ref;
+	int			instance;
 };
 
 struct nvme_ns {
@@ -244,7 +257,6 @@ struct nvme_ns {
 	struct nvm_dev *ndev;
 	struct kref kref;
 	struct nvme_ns_head *head;
-	int instance;
 
 	int lba_shift;
 	u16 ms;
-- 
2.14.1



From hch at lst.de  Wed Oct 18 09:52:58 2017
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 18 Oct 2017 18:52:58 +0200
Subject: [PATCH 17/17] nvme: also expose the namespace identification sysfs
 files for mpath nodes
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <20171018165258.23212-18-hch@lst.de>

We do this by adding a helper that returns the ns_head for a device that
can belong to either the per-controller or per-subsystem block device
nodes, and otherwise reuse all the existing code.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 67 +++++++++++++++++++++++++++++-------------------
 1 file changed, 41 insertions(+), 26 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 49a81118c9f8..529a1cf97e61 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -76,6 +76,7 @@ static DEFINE_IDA(nvme_instance_ida);
 static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 static struct class *nvme_subsys_class;
+static const struct attribute_group nvme_ns_id_attr_group;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
 {
@@ -329,6 +330,8 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	sysfs_remove_group(&disk_to_dev(head->disk)->kobj,
+			   &nvme_ns_id_attr_group);
 	del_gendisk(head->disk);
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
@@ -2267,12 +2270,22 @@ static ssize_t nvme_sysfs_rescan(struct device *dev,
 }
 static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan);
 
+static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
+{
+	struct gendisk *disk = dev_to_disk(dev);
+
+	if (disk->fops == &nvme_fops)
+		return nvme_get_ns_from_dev(dev)->head;
+	else
+		return disk->private_data;
+}
+
 static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
-	struct nvme_subsystem *subsys = ns->ctrl->subsys;
+	struct nvme_ns_head *head = dev_to_ns_head(dev);
+	struct nvme_ns_ids *ids = &head->ids;
+	struct nvme_subsystem *subsys = head->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
 
@@ -2294,23 +2307,21 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 
 	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
 		serial_len, subsys->serial, model_len, subsys->model,
-		ns->head->ns_id);
+		head->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
 static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
-			  char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->head->ids.nguid);
+	return sprintf(buf, "%pU\n", dev_to_ns_head(dev)->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
 static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
@@ -2325,22 +2336,20 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
 
 static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->head->ids.eui64);
+	return sprintf(buf, "%8phd\n", dev_to_ns_head(dev)->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
 static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%d\n", ns->head->ns_id);
+	return sprintf(buf, "%d\n", dev_to_ns_head(dev)->ns_id);
 }
 static DEVICE_ATTR(nsid, S_IRUGO, nsid_show, NULL);
 
-static struct attribute *nvme_ns_attrs[] = {
+static struct attribute *nvme_ns_id_attrs[] = {
 	&dev_attr_wwid.attr,
 	&dev_attr_uuid.attr,
 	&dev_attr_nguid.attr,
@@ -2349,12 +2358,11 @@ static struct attribute *nvme_ns_attrs[] = {
 	NULL,
 };
 
-static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
+static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
 	if (a == &dev_attr_uuid.attr) {
 		if (uuid_is_null(&ids->uuid) ||
@@ -2372,9 +2380,9 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 	return a->mode;
 }
 
-static const struct attribute_group nvme_ns_attr_group = {
-	.attrs		= nvme_ns_attrs,
-	.is_visible	= nvme_ns_attrs_are_visible,
+static const struct attribute_group nvme_ns_id_attr_group = {
+	.attrs		= nvme_ns_id_attrs,
+	.is_visible	= nvme_ns_id_attrs_are_visible,
 };
 
 #define nvme_show_str_function(field)						\
@@ -2851,15 +2859,20 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	device_add_disk(ctrl->device, ns->disk);
 	if (sysfs_create_group(&disk_to_dev(ns->disk)->kobj,
-					&nvme_ns_attr_group))
+					&nvme_ns_id_attr_group))
 		pr_warn("%s: failed to create sysfs group for identification\n",
 			ns->disk->disk_name);
 	if (ns->ndev && nvme_nvm_register_sysfs(ns))
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
 
-	if (new)
+	if (new) {
 		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
+		if (sysfs_create_group(&disk_to_dev(ns->head->disk)->kobj,
+				&nvme_ns_id_attr_group))
+			pr_warn("%s: failed to create sysfs group for identification\n",
+				ns->head->disk->disk_name);
+	}
 
 	return;
  out_unlink_ns:
@@ -2884,6 +2897,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		if (blk_get_integrity(ns->disk))
 			blk_integrity_unregister(ns->disk);
+		sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
+				   &nvme_ns_id_attr_group);
 		if (ns->ndev)
 			nvme_nvm_unregister_sysfs(ns);
 		del_gendisk(ns->disk);
-- 
2.14.1



From maxg at mellanox.com  Wed Oct 18 13:05:16 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Wed, 18 Oct 2017 23:05:16 +0300
Subject: [Suspected-Phishing][PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to
 admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <a179f3ae-7b4c-da5a-365d-abe15e02372a@mellanox.com>



On 10/18/2017 3:38 PM, Israel Rukshin wrote:
> Since commit b86dd81
> "block: get rid of blk-mq default scheduler choice Kconfig entries",
> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
> This flag is useful for admin queues that aren't used for normal IO.
> We already use this flag at nvme-pci.
> 
> Israel Rukshin (3):
>    nvme-rdma: Add BLK_MQ_F_NO_SCHED flag to admin tag set
>    nvme-fc: Add BLK_MQ_F_NO_SCHED flag to admin tag set
>    nvme-loop: Add BLK_MQ_F_NO_SCHED flag to admin tag set

thanks Israel.
all 3 patches looks good to me,

Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From keith.busch at intel.com  Wed Oct 18 13:17:13 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 14:17:13 -0600
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171018150859.GD32530@localhost.localdomain>
 <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>
Message-ID: <20171018201713.GA1971@localhost.localdomain>

On Wed, Oct 18, 2017 at 06:29:40PM +0300, Sagi Grimberg wrote:
> 
> > > From: Roy Shterman <roys at lightbitslabs.com>
> > > 
> > > Its strange to have some of the cleanup for nvme_init_ctrl in
> > > nvme_free_ctrl. This cause us to have some magic ctrl refcount
> > > handling in controller initialization error paths.
> > > 
> > > Make the init/uninit symmetric so we can have a sane and
> > > maintainable error sequences.
> > 
> > It's not magic. It's just preventing name reuse when some management
> > code retains an open handle after the device was removed.
> 
> management code as in nvme-cli?
> 
> > Otherwise
> > you're going to get naming clashes when you bring up a new device with
> > the same name from reusing the still in-use instance.
> 
> Can userspace keep an open-handle after device_destroy() was invoked?
> 
> Can you explain how would I see naming clashes? I'm not sure I
> understand what you are referring to.

Hm, I'm not hitting duplicate name issues anymore. I can't be bothered
to bisect when that started working, but I think the dev_t lifetime
fixes has something to do with it.

Anyway, simpler is certainly preferred. This is looking okay so far,
and I'll try a few more things to see if I can break it.


From keith.busch at intel.com  Wed Oct 18 13:31:47 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 14:31:47 -0600
Subject: [PATCH] nvme-cli : wdc-plugin Add support for WDC SN100 and
 SN200 devices.
In-Reply-To: <20171017165643.31988-1-jeff.lien@hgst.com>
References: <20171017165643.31988-1-jeff.lien@hgst.com>
Message-ID: <20171018203147.GA2040@localhost.localdomain>

On Tue, Oct 17, 2017 at 11:56:43AM -0500, jeffreyalien wrote:
> From: jeffreyalien <jeff.lien at wdc.com>
> 
> Update the checking for the wdc-plugin commands so that a
> "Device not supported" message will be displayed for non-WDC
> devices.
> 
> Signed-off-by: jeffreyalien <jeff.lien at wdc.com>

Thanks, applied.


From brking at linux.vnet.ibm.com  Wed Oct 18 13:57:28 2017
From: brking at linux.vnet.ibm.com (Brian King)
Date: Wed, 18 Oct 2017 15:57:28 -0500
Subject: [PATCH v2 6/8] scsi/ipr: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <c804098f-5543-395c-1395-5778ef74a61b@suse.de>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-7-bart.vanassche@wdc.com>
 <c804098f-5543-395c-1395-5778ef74a61b@suse.de>
Message-ID: <ce6f2f5b-b4bc-380f-8ac1-1e91e4d6bf24@linux.vnet.ibm.com>

On 10/17/2017 01:19 AM, Hannes Reinecke wrote:
> On 10/17/2017 12:49 AM, Bart Van Assche wrote:
>> Use the sgl_alloc_order() and sgl_free_order() functions instead
>> of open coding these functions.
>>
>> Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
>> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
>> Cc: linux-scsi at vger.kernel.org
>> Cc: Martin K. Petersen <martin.petersen at oracle.com>
>> Cc: Brian King <brking at linux.vnet.ibm.com>
>> ---
>>  drivers/scsi/Kconfig |  1 +
>>  drivers/scsi/ipr.c   | 49 ++++++++-----------------------------------------
>>  drivers/scsi/ipr.h   |  2 +-
>>  3 files changed, 10 insertions(+), 42 deletions(-)
>>
>> diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
>> index 41366339b950..d11e75e76195 100644
>> --- a/drivers/scsi/Kconfig
>> +++ b/drivers/scsi/Kconfig
>> @@ -1058,6 +1058,7 @@ config SCSI_IPR
>>  	depends on PCI && SCSI && ATA
>>  	select FW_LOADER
>>  	select IRQ_POLL
>> +	select SGL_ALLOC
>>  	---help---
>>  	  This driver supports the IBM Power Linux family RAID adapters.
>>  	  This includes IBM pSeries 5712, 5703, 5709, and 570A, as well
>> diff --git a/drivers/scsi/ipr.c b/drivers/scsi/ipr.c
>> index f838bd73befa..6fed5db6255e 100644
>> --- a/drivers/scsi/ipr.c
>> +++ b/drivers/scsi/ipr.c
>> @@ -3815,10 +3815,8 @@ static struct device_attribute ipr_iopoll_weight_attr = {
>>   **/
>>  static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>>  {
>> -	int sg_size, order, bsize_elem, num_elem, i, j;
>> +	int sg_size, order;
>>  	struct ipr_sglist *sglist;
>> -	struct scatterlist *scatterlist;
>> -	struct page *page;
>>  
>>  	/* Get the minimum size per scatter/gather element */
>>  	sg_size = buf_len / (IPR_MAX_SGLIST - 1);
>> @@ -3826,45 +3824,18 @@ static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>>  	/* Get the actual size per element */
>>  	order = get_order(sg_size);
>>  
>> -	/* Determine the actual number of bytes per element */
>> -	bsize_elem = PAGE_SIZE * (1 << order);
>> -
>> -	/* Determine the actual number of sg entries needed */
>> -	if (buf_len % bsize_elem)
>> -		num_elem = (buf_len / bsize_elem) + 1;
>> -	else
>> -		num_elem = buf_len / bsize_elem;
>> -
>>  	/* Allocate a scatter/gather list for the DMA */
>> -	sglist = kzalloc(sizeof(struct ipr_sglist) +
>> -			 (sizeof(struct scatterlist) * (num_elem - 1)),
>> -			 GFP_KERNEL);
>> -
>> +	sglist = kzalloc(sizeof(struct ipr_sglist), GFP_KERNEL);
>>  	if (sglist == NULL) {
>>  		ipr_trace;
>>  		return NULL;
>>  	}
>> -
>> -	scatterlist = sglist->scatterlist;
>> -	sg_init_table(scatterlist, num_elem);
>> -
>>  	sglist->order = order;
>> -	sglist->num_sg = num_elem;
>> -
>> -	/* Allocate a bunch of sg elements */
>> -	for (i = 0; i < num_elem; i++) {
>> -		page = alloc_pages(GFP_KERNEL, order);
>> -		if (!page) {
>> -			ipr_trace;
>> -
>> -			/* Free up what we already allocated */
>> -			for (j = i - 1; j >= 0; j--)
>> -				__free_pages(sg_page(&scatterlist[j]), order);
>> -			kfree(sglist);
>> -			return NULL;
>> -		}
>> -
>> -		sg_set_page(&scatterlist[i], page, 0, 0);
>> +	sglist->scatterlist = sgl_alloc_order(buf_len, order, false, GFP_KERNEL,
>> +					      &sglist->num_sg);
>> +	if (!sglist->scatterlist) {
>> +		kfree(sglist);
>> +		return NULL;
>>  	}
>>  
>>  	return sglist;
>> @@ -3882,11 +3853,7 @@ static struct ipr_sglist *ipr_alloc_ucode_buffer(int buf_len)
>>   **/
>>  static void ipr_free_ucode_buffer(struct ipr_sglist *sglist)
>>  {
>> -	int i;
>> -
>> -	for (i = 0; i < sglist->num_sg; i++)
>> -		__free_pages(sg_page(&sglist->scatterlist[i]), sglist->order);
>> -
>> +	sgl_free_order(sglist->scatterlist, sglist->order);
>>  	kfree(sglist);
>>  }
>>  
>> diff --git a/drivers/scsi/ipr.h b/drivers/scsi/ipr.h
>> index c7f0e9e3cd7d..93570734cbfb 100644
>> --- a/drivers/scsi/ipr.h
>> +++ b/drivers/scsi/ipr.h
>> @@ -1454,7 +1454,7 @@ struct ipr_sglist {
>>  	u32 num_sg;
>>  	u32 num_dma_sg;
>>  	u32 buffer_len;
>> -	struct scatterlist scatterlist[1];
>> +	struct scatterlist *scatterlist;
>>  };
>>  
>>  enum ipr_sdt_state {
>>
> Not sure if this is a valid conversion.
> Originally the driver would allocate a single buffer; with this buffer
> we have two distinct buffers.
> Given that this is used to download the microcode I'm not sure if this
> isn't a hardware-dependent structure which requires a single buffer
> including the sglist.
> Brian, can you shed some light here?

The struct ipr_sglist is not a hardware defined data structure, so on initial
glance, this should be OK. I'll load it up and give it a try to make sure
it doesn't break code download.

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center



From jsmart2021 at gmail.com  Wed Oct 18 14:33:59 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 18 Oct 2017 14:33:59 -0700
Subject: [PATCH v4] nvmet: synchronize sqhd update
Message-ID: <20171018213359.3587-1-jsmart2021@gmail.com>

In testing target io in read write mix, we did indeed get into cases
where sqhd didn't update properly and slowly missed enough updates to
shutdown the queue.

Protect the updating sqhd by using cmpxchg

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2: move locks so around update and assignment
v3: convert to cmpxchg
v4: as u16 cmpxchg appears to not always be supported, convert sqhd to u32
---
 drivers/nvme/target/core.c  | 15 ++++++++++++---
 drivers/nvme/target/nvmet.h |  2 +-
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index cfc35cbb6fe2..a469f1996dca 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -387,12 +387,21 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 static void __nvmet_req_complete(struct nvmet_req *req, u16 status)
 {
+	u32 old_sqhd, new_sqhd;
+	u16 sqhd;
+
 	if (status)
 		nvmet_set_status(req, status);
 
-	if (req->sq->size)
-		req->sq->sqhd = (req->sq->sqhd + 1) % req->sq->size;
-	req->rsp->sq_head = cpu_to_le16(req->sq->sqhd);
+	if (req->sq->size) {
+		do {
+			old_sqhd = req->sq->sqhd;
+			new_sqhd = (old_sqhd + 1) % req->sq->size;
+		} while (cmpxchg(&req->sq->sqhd, old_sqhd, new_sqhd) !=
+					old_sqhd);
+	}
+	sqhd = req->sq->sqhd & 0x0000FFFF;
+	req->rsp->sq_head = cpu_to_le16(sqhd);
 	req->rsp->sq_id = cpu_to_le16(req->sq->qid);
 	req->rsp->command_id = req->cmd->common.command_id;
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index ed38b44a7007..ebc1446e205e 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -74,7 +74,7 @@ struct nvmet_sq {
 	struct percpu_ref	ref;
 	u16			qid;
 	u16			size;
-	u16			sqhd;
+	u32			sqhd;
 	struct completion	free_done;
 	struct completion	confirm_done;
 };
-- 
2.13.1



From james.smart at broadcom.com  Wed Oct 18 14:41:37 2017
From: james.smart at broadcom.com (James Smart)
Date: Wed, 18 Oct 2017 14:41:37 -0700
Subject: [PATCH 2/3] nvme-fc: Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-3-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
 <1508330306-26844-3-git-send-email-israelr@mellanox.com>
Message-ID: <d0e51f56-401c-b1cb-90ad-a3cc8e208bd4@broadcom.com>

On 10/18/2017 5:38 AM, Israel Rukshin wrote:
> Since commit b86dd81
> "block: get rid of blk-mq default scheduler choice Kconfig entries",
> when setting nr_hw_queues to 1 the admin tag set uses mq-deadline scheduler.
> This flag is useful for admin queues that aren't used for normal IO.
>
> Signed-off-by: Israel Rukshin <israelr at mellanox.com>
> ---
>   drivers/nvme/host/fc.c | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index af075e9..b383735 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2796,6 +2796,7 @@ enum {
>   	ctrl->admin_tag_set.driver_data = ctrl;
>   	ctrl->admin_tag_set.nr_hw_queues = 1;
>   	ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT;
> +	ctrl->admin_tag_set.flags = BLK_MQ_F_NO_SCHED;
>   
>   	ret = blk_mq_alloc_tag_set(&ctrl->admin_tag_set);
>   	if (ret)

Looks fine

Signed-off-by: James Smart? <james.smart at broadcom.com>



From keith.busch at intel.com  Wed Oct 18 15:39:13 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 16:39:13 -0600
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171018165258.23212-14-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-14-hch@lst.de>
Message-ID: <20171018223913.GA1846@localhost.localdomain>

On Wed, Oct 18, 2017 at 06:52:54PM +0200, Christoph Hellwig wrote:
> +
> +static void nvme_put_subsystem(struct nvme_subsystem *subsys)
> +{
> +	put_device(&subsys->dev);
> +}

<snip>

> +static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
> +{
> +	struct nvme_subsystem *subsys, *found;
> +	int ret;
> +
> +	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
> +	if (!subsys)
> +		return -ENOMEM;
> +	INIT_LIST_HEAD(&subsys->ctrls);
> +	nvme_init_subnqn(subsys, ctrl, id);
> +	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
> +	memcpy(subsys->model, id->mn, sizeof(subsys->model));
> +	memcpy(subsys->firmware_rev, id->fr, sizeof(subsys->firmware_rev));
> +	subsys->vendor_id = le16_to_cpu(id->vid);
> +	mutex_init(&subsys->lock);
> +
> +	subsys->dev.class = nvme_subsys_class;
> +	subsys->dev.release = nvme_free_subsystem;
> +	dev_set_name(&subsys->dev, subsys->subnqn);
> +	device_initialize(&subsys->dev);
> +
> +	mutex_lock(&nvme_subsystems_lock);
> +	found = __nvme_find_get_subsystem(subsys->subnqn);
> +	if (found) {
> +		/*
> +		 * Verify that the subsystem actually supports multiple
> +		 * controllers, else bail out.
> +		 */
> +		if (!(id->cmic & (1 << 1))) {
> +			dev_err(ctrl->device,
> +				"ignoring ctrl due to duplicate subnqn (%s).\n",
> +				found->subnqn);
> +			nvme_put_subsystem(found);
> +			ret = -EINVAL;
> +			goto out_unlock;
> +		}
> +
> +		kfree(subsys);
> +		subsys = found;
> +	} else {
> +		ret = device_add(&subsys->dev);
> +		if (ret) {
> +			dev_err(ctrl->device,
> +				"failed to register subsystem device.\n");
> +			goto out_unlock;
> +		}
> +		list_add_tail(&subsys->entry, &nvme_subsystems);
> +	}

I'm having some trouble with this one. The device_initialize() initializes
the kobj's reference counter, and then the device_add() takes another
reference on it.

The teardown, though, only calls put_device(). Where's the call to
device_del() supposed to go that ultimately drops the last reference
to call the .release 'nvme_free_subsystem'?


From keith.busch at intel.com  Wed Oct 18 15:53:56 2017
From: keith.busch at intel.com (Keith Busch)
Date: Wed, 18 Oct 2017 16:53:56 -0600
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171018223913.GA1846@localhost.localdomain>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-14-hch@lst.de>
 <20171018223913.GA1846@localhost.localdomain>
Message-ID: <20171018225356.GA1796@localhost.localdomain>

With the series, 'modprobe -r nvme && modprobe nvme' is failing. I can
get that to pass with the following:

---
@@ -1904,7 +1907,11 @@ static void nvme_free_subsystem(struct device *dev)
 static void nvme_put_subsystem(struct nvme_subsystem *subsys)
 {
 	put_device(&subsys->dev);
+
+	if (list_empty(&subsys->ctrls))
+		device_del(&subsys->dev);
 }
 
 static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
--

But this is racy with nvme_init_subsystem, and it creates warnings:

  kernfs: can not remove 'uevent', no directory
  kernfs: can not remove 'online', no directory


From daniel.verkamp at intel.com  Wed Oct 18 16:52:04 2017
From: daniel.verkamp at intel.com (Verkamp, Daniel)
Date: Wed, 18 Oct 2017 23:52:04 +0000
Subject: NVMe target log page offset not supported
Message-ID: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>

Hello,

It looks like the Linux NVMe over Fabrics target Get Log Page implementation is missing support for the Log Page Offset field, which is part of the optional "extended data for Get Log Page" feature.

Of the two Get Log Page implementations, in drivers/nvme/target/{admin-cmd.c,discovery.c}, both advertise support for extended data for Get Log Page in the Identify Controller LPA field (bit 2), but neither looks at the offset field (CDW12 and CDW13, named LPOL and LPOU in the spec).

In theory, it is tempting to just stop advertising the extended log page feature, but reading between the lines in the NVMe-oF 1.0 spec, it seems like this might be intended to be a required feature for at least discovery controllers (e.g. the Discovery Log Page section says "The Log Page Offset is used (if needed) to retrieve specific records").  Maybe the feature should be temporarily disabled anyway until the offset is correctly implemented?

It also looks like the discovery.c Get Log Page implementation has some funny ideas about the meaning of the log page length (Number of Dwords field).  A request with a length that doesn't fully cover a discovery log entry will not return the partial data as requested but will instead stop at the last full entry and skip the entire final partial entry, leaving it as zeroes.  To be fair, as is usual with NVMe, the spec doesn't actually say what to do in this situation (it only defines expected behavior when the user requests more than the total length of the log page), but it seems like the right thing to do would be to return exactly the [offset, offset + length) bytes of the log page, even if that range overlaps the beginning/end of a discovery entry (or the header, which can happen once offset is implemented).

Unfortunately, I don't have time right now to write and thoroughly test a patch for this, so I thought I'd at least make it known.  It's on my todo list...

Thanks,
-- Daniel


From hare at suse.de  Wed Oct 18 22:48:02 2017
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 19 Oct 2017 07:48:02 +0200
Subject: [PATCH 01/17] block: move REQ_NOWAIT
In-Reply-To: <20171018165258.23212-2-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-2-hch@lst.de>
Message-ID: <8895468d-a36a-9f9c-0086-22e058cf5b23@suse.de>

On 10/18/2017 06:52 PM, Christoph Hellwig wrote:
> This flag should be before the operation-specific REQ_NOUNMAP bit.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>  include/linux/blk_types.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Wed Oct 18 22:48:29 2017
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 19 Oct 2017 07:48:29 +0200
Subject: [PATCH 02/17] block: add REQ_DRV bit
In-Reply-To: <20171018165258.23212-3-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-3-hch@lst.de>
Message-ID: <74620ec2-91ed-b76d-4bc3-46b5f5e03e14@suse.de>

On 10/18/2017 06:52 PM, Christoph Hellwig wrote:
> Set aside a bit in the request/bio flags for driver use.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>  include/linux/blk_types.h | 5 +++++
>  1 file changed, 5 insertions(+)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From yizhan at redhat.com  Wed Oct 18 23:33:46 2017
From: yizhan at redhat.com (Yi Zhang)
Date: Thu, 19 Oct 2017 14:33:46 +0800
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
 <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
Message-ID: <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>



On 10/16/2017 06:21 PM, Sagi Grimberg wrote:
>>> Lets take this one step at a time, starting with this issue.
>>>
>>> First, there is a reason why a simple create_ctrl fails, can we isolate
>>> exactly which call fails? Was something else going on that might have
>>> made the simple create_ctrl fail?
>>>
>>> We don't see any "rdma_resolve_addr failed" or "failed to connect 
>>> queue"
>>> messages but we do see "creating I/O queues" which means that we either
>>> failed at IO tagset allocation or initializing connect_q.
>>>
>>> We have a missing error code assignment so can you try the following 
>>> patch:
>>
>> Hi Sagi
>> Sorry for the late response, here is the log with your patch, let me 
>> know if you need more test.
>
> Hi Yi,
>
> I was referring to the bug you reported on a simple create_ctrl failed:
> https://pastebin.com/7z0XSGSd
>
> Does it reproduce?
>
yes, this issue was reproduced during "git bisect" with below patch

commit 31fdf18401703df246e20fc0382ed3594e6cf8d8
Author: Sagi Grimberg <sagi at grimberg.me>
Date:   Mon Aug 28 21:40:06 2017 +0200

     nvme-rdma: reuse configure/destroy_admin_queue

     No need to open-code it.

     Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
     Signed-off-by: Christoph Hellwig <hch at lst.de>

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme



From jthumshirn at suse.de  Wed Oct 18 23:46:53 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 08:46:53 +0200
Subject: [PATCH 01/17] block: move REQ_NOWAIT
In-Reply-To: <20171018165258.23212-2-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:42 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-2-hch@lst.de>
Message-ID: <mqd4lqv8wiq.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 23:47:32 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 08:47:32 +0200
Subject: [PATCH 02/17] block: add REQ_DRV bit
In-Reply-To: <20171018165258.23212-3-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:43 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-3-hch@lst.de>
Message-ID: <mqdy3o77hx7.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 23:53:07 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 08:53:07 +0200
Subject: [PATCH 07/17] nvme: use ida_simple_{get,
 remove} for the controller instance
In-Reply-To: <20171018165258.23212-8-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:48 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-8-hch@lst.de>
Message-ID: <mqdshef7hnw.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 23:54:08 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 08:54:08 +0200
Subject: [PATCH 08/17] nvme: use kref_get_unless_zero in nvme_find_get_ns
In-Reply-To: <20171018165258.23212-9-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:49 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-9-hch@lst.de>
Message-ID: <mqdmv4n7hm7.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Wed Oct 18 23:55:07 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 08:55:07 +0200
Subject: [PATCH 09/17] nvme: simplify nvme_open
In-Reply-To: <20171018165258.23212-10-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:50 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-10-hch@lst.de>
Message-ID: <mqdh8uv7hkk.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Wed Oct 18 23:55:46 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 09:55:46 +0300
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
 <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
 <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>
Message-ID: <62eea88b-caa4-5799-3d8f-8d8789879aa8@grimberg.me>


>> Hi Yi,
>>
>> I was referring to the bug you reported on a simple create_ctrl failed:
>> https://pastebin.com/7z0XSGSd
>>
>> Does it reproduce?
>>
> yes, this issue was reproduced during "git bisect" with below patch

OK, if this does not reproduce with the latest code, lets put it aside
for now.

So as for the error you see, can you please try the following patch?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 405895b1dff2..916658e010ff 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct 
nvme_rdma_queue *queue)
         if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
                 return;

+       if(nvme_rdma_queue_idx(queue) == 0)
+               nvme_rdma_free_qe(queue->device->dev,
+                       &queue->ctrl->async_event_sqe,
+                       sizeof(struct nvme_command), DMA_TO_DEVICE);
+
         nvme_rdma_destroy_queue_ib(queue);
         rdma_destroy_id(queue->cm_id);
  }
@@ -739,8 +744,6 @@ static struct blk_mq_tag_set 
*nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
  static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
                 bool remove)
  {
-       nvme_rdma_free_qe(ctrl->queues[0].device->dev, 
&ctrl->async_event_sqe,
-                       sizeof(struct nvme_command), DMA_TO_DEVICE);
         nvme_rdma_stop_queue(&ctrl->queues[0]);
         if (remove) {
                 blk_cleanup_queue(ctrl->ctrl.admin_q);
--


From sagi at grimberg.me  Wed Oct 18 23:58:09 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 09:58:09 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
Message-ID: <1508396289-12237-1-git-send-email-sagi@grimberg.me>

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 405895b1dff2..4948c0fb3d05 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -765,8 +765,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 
 	if (new) {
 		ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
-		if (IS_ERR(ctrl->ctrl.admin_tagset))
+		if (IS_ERR(ctrl->ctrl.admin_tagset)) {
+			error = PTR_ERR(ctrl->ctrl.admin_tagset);
 			goto out_free_queue;
+		}
 
 		ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set);
 		if (IS_ERR(ctrl->ctrl.admin_q)) {
@@ -845,8 +847,10 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
 	if (new) {
 		ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, false);
-		if (IS_ERR(ctrl->ctrl.tagset))
+		if (IS_ERR(ctrl->ctrl.tagset)) {
+			ret = PTR_ERR(ctrl->ctrl.tagset);
 			goto out_free_io_queues;
+		}
 
 		ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set);
 		if (IS_ERR(ctrl->ctrl.connect_q)) {
-- 
2.7.4



From sagi at grimberg.me  Wed Oct 18 23:58:48 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 09:58:48 +0300
Subject: [PATCH 07/17] nvme: use ida_simple_{get,remove} for the
 controller instance
In-Reply-To: <20171018165258.23212-8-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-8-hch@lst.de>
Message-ID: <5652652c-8fa4-b346-b85e-e6dfcf16c9b1@grimberg.me>

Looks much better,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 18 23:59:21 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 09:59:21 +0300
Subject: [PATCH 08/17] nvme: use kref_get_unless_zero in nvme_find_get_ns
In-Reply-To: <20171018165258.23212-9-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-9-hch@lst.de>
Message-ID: <874a72f9-3982-ca48-9d35-0600068a1ad0@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Oct 18 23:59:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 09:59:42 +0300
Subject: [PATCH 09/17] nvme: simplify nvme_open
In-Reply-To: <20171018165258.23212-10-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-10-hch@lst.de>
Message-ID: <699496c3-377a-397a-1d78-6c1021e1a456@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From hch at lst.de  Thu Oct 19 00:14:05 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 09:14:05 +0200
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171018223913.GA1846@localhost.localdomain>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-14-hch@lst.de>
 <20171018223913.GA1846@localhost.localdomain>
Message-ID: <20171019071405.GA11980@lst.de>

> I'm having some trouble with this one. The device_initialize() initializes
> the kobj's reference counter, and then the device_add() takes another
> reference on it.
> 
> The teardown, though, only calls put_device(). Where's the call to
> device_del() supposed to go that ultimately drops the last reference
> to call the .release 'nvme_free_subsystem'?

Yes, this is buggy.  I took the code from Hannes patches without
retesting a module unload.  I'll go back to the drawing board.


From sagi at grimberg.me  Thu Oct 19 00:17:01 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 10:17:01 +0300
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct
 device
In-Reply-To: <20171018165258.23212-11-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
Message-ID: <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>



> -	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
> -		return -EBUSY;
> +	nvme_get_ctrl(&ctrl->ctrl);

Given that we take this reference before we are protected with
the state change I think this should still be get_unless_zero.

Maybe we can introduce get_device_unless_zero() for this?


From sagi at grimberg.me  Thu Oct 19 00:18:10 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 10:18:10 +0300
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <20171018165258.23212-12-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-12-hch@lst.de>
Message-ID: <acd8acfe-5f9a-6b1e-67ff-22a7ba8c16c3@grimberg.me>

Looks good!

Reviewed-by: Sagi Grimberg <sgi at grimberg.me>


From sagi at grimberg.me  Thu Oct 19 00:18:28 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 10:18:28 +0300
Subject: [PATCH 12/17] nvme: check for a live controller in nvme_dev_open
In-Reply-To: <20171018165258.23212-13-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-13-hch@lst.de>
Message-ID: <562df233-f146-66fd-f660-fb773884d36e@grimberg.me>

Looks good,

Reviewed-by: Sagi Grimberg <sagi at rimbeg.me>


From hch at lst.de  Thu Oct 19 00:20:53 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 09:20:53 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use
 struct device
In-Reply-To: <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
Message-ID: <20171019072053.GA12168@lst.de>

On Thu, Oct 19, 2017 at 10:17:01AM +0300, Sagi Grimberg wrote:
>
>
>> -	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
>> -		return -EBUSY;
>> +	nvme_get_ctrl(&ctrl->ctrl);
>
> Given that we take this reference before we are protected with
> the state change I think this should still be get_unless_zero.

Because we now refcount the device we must have a reference on it
when we call the sysfs file for it, and for the fabrics file
we have an explicit reference already.  So there should not be
any need to do the unless_zero.

> Maybe we can introduce get_device_unless_zero() for this?

I thought about it for the other open coded versions and my decision
was that I want to send a patch for it next merge window, but not
now to avoid having to coordinate with the driver core tree.


From jthumshirn at suse.de  Thu Oct 19 00:22:09 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 09:22:09 +0200
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <20171018165258.23212-12-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:52 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-12-hch@lst.de>
Message-ID: <mqd60bb7gbi.fsf@linux-x5ow.site>

I'm not sure if the removel of the major number module_param is good
from an ABI POV. This could lead to some unexpected results for those
poor souls that used it.

Apart from that,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Thu Oct 19 00:23:49 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 09:23:49 +0200
Subject: [PATCH 12/17] nvme: check for a live controller in nvme_dev_open
In-Reply-To: <20171018165258.23212-13-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:53 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-13-hch@lst.de>
Message-ID: <mqdvajb61oa.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Thu Oct 19 00:24:00 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 09:24:00 +0200
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <mqd60bb7gbi.fsf@linux-x5ow.site>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-12-hch@lst.de> <mqd60bb7gbi.fsf@linux-x5ow.site>
Message-ID: <20171019072400.GA12233@lst.de>

On Thu, Oct 19, 2017 at 09:22:09AM +0200, Johannes Thumshirn wrote:
> I'm not sure if the removel of the major number module_param is good
> from an ABI POV. This could lead to some unexpected results for those
> poor souls that used it.

It's a completely broken interface, so I have no pity for them..


From sagi at grimberg.me  Thu Oct 19 00:31:20 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 10:31:20 +0300
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct
 device
In-Reply-To: <20171019072053.GA12168@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
Message-ID: <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>


>>> -	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
>>> -		return -EBUSY;
>>> +	nvme_get_ctrl(&ctrl->ctrl);
>>
>> Given that we take this reference before we are protected with
>> the state change I think this should still be get_unless_zero.
> 
> Because we now refcount the device we must have a reference on it
> when we call the sysfs file for it, and for the fabrics file
> we have an explicit reference already.  So there should not be
> any need to do the unless_zero.

I don't think the fabrics device does not help us to keep the ctrl
allocated until we finish removal.

I do agree that the kobj reference in nvme_dev_open keeps it alive.


From jthumshirn at suse.de  Thu Oct 19 00:36:02 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 09:36:02 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171018165258.23212-16-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:56 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-16-hch@lst.de>
Message-ID: <mqdk1zr613x.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Thu Oct 19 00:37:41 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 09:37:41 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use
 struct device
In-Reply-To: <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
 <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
Message-ID: <20171019073741.GA12348@lst.de>

On Thu, Oct 19, 2017 at 10:31:20AM +0300, Sagi Grimberg wrote:
> I don't think the fabrics device does not help us to keep the ctrl
> allocated until we finish removal.

All fabrics drivers grab an extra reference during ->create_ctrl,
which we will drop when releasing the file descriptor that we used
to create the device.

By the time we call ->delete_ctrl in nvmf_create_ctrl we must still
have that reference because we are still in a ->write call and thus
->release can't have been called yet.


From jthumshirn at suse.de  Thu Oct 19 00:45:56 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 09:45:56 +0200
Subject: [PATCH 17/17] nvme: also expose the namespace identification
 sysfs files for mpath nodes
In-Reply-To: <20171018165258.23212-18-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:58 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-18-hch@lst.de>
Message-ID: <mqdd15j60nf.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Thu Oct 19 01:00:01 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 10:00:01 +0200
Subject: [PATCH v3] nvmet: synchronize sqhd update
In-Reply-To: <20171018165216.25563-1-jsmart2021@gmail.com> (James Smart's
 message of "Wed, 18 Oct 2017 09:52:16 -0700")
References: <20171018165216.25563-1-jsmart2021@gmail.com>
Message-ID: <mqd60bb5zzy.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From yi.zhang at redhat.com  Thu Oct 19 01:23:38 2017
From: yi.zhang at redhat.com (Yi Zhang)
Date: Thu, 19 Oct 2017 16:23:38 +0800
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <62eea88b-caa4-5799-3d8f-8d8789879aa8@grimberg.me>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
 <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
 <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>
 <62eea88b-caa4-5799-3d8f-8d8789879aa8@grimberg.me>
Message-ID: <0561ed58-7104-d8b6-1709-ea181ac493f5@redhat.com>



On 10/19/2017 02:55 PM, Sagi Grimberg wrote:
>
>>> Hi Yi,
>>>
>>> I was referring to the bug you reported on a simple create_ctrl failed:
>>> https://pastebin.com/7z0XSGSd
>>>
>>> Does it reproduce?
>>>
>> yes, this issue was reproduced during "git bisect" with below patch
>
> OK, if this does not reproduce with the latest code, lets put it aside
> for now.
>
> So as for the error you see, can you please try the following patch?
Hi Sagi
With this patch, no such error log found on host side, but I found there 
is no nvme0n1 device node even get "nvme nvme0: Successfully 
reconnected" on host.

Host side:
[   98.181089] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[   98.329464] nvme nvme0: creating 40 I/O queues.
[   98.835409] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  107.873586] nvme nvme0: Reconnecting in 10 seconds...
[  118.505937] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  118.513443] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  118.519875] nvme nvme0: Failed reconnect attempt 1
[  118.525241] nvme nvme0: Reconnecting in 10 seconds...
[  128.733311] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  128.740812] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  128.747247] nvme nvme0: Failed reconnect attempt 2
[  128.752609] nvme nvme0: Reconnecting in 10 seconds...
[  138.973404] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  138.980904] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  138.987329] nvme nvme0: Failed reconnect attempt 3
[  138.992691] nvme nvme0: Reconnecting in 10 seconds...
[  149.232610] nvme nvme0: creating 40 I/O queues.
[  149.831443] nvme nvme0: Successfully reconnected
[  149.831519] nvme nvme0: identifiers changed for nsid 1
[root at rdma-virt-01 linux ((dafb1b2...))]$ lsblk
NAME                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                              8:0    0 465.8G  0 disk
??sda2                           8:2    0 464.8G  0 part
? ??rhelaa_rdma--virt--01-swap 253:1    0     4G  0 lvm  [SWAP]
? ??rhelaa_rdma--virt--01-home 253:2    0 410.8G  0 lvm  /home
? ??rhelaa_rdma--virt--01-root 253:0    0    50G  0 lvm  /
??sda1                           8:1    0     1G  0 part /boot

> -- 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 405895b1dff2..916658e010ff 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct 
> nvme_rdma_queue *queue)
>         if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
>                 return;
>
> +       if(nvme_rdma_queue_idx(queue) == 0)
> +               nvme_rdma_free_qe(queue->device->dev,
> +                       &queue->ctrl->async_event_sqe,
> +                       sizeof(struct nvme_command), DMA_TO_DEVICE);
> +
>         nvme_rdma_destroy_queue_ib(queue);
>         rdma_destroy_id(queue->cm_id);
>  }
> @@ -739,8 +744,6 @@ static struct blk_mq_tag_set 
> *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
>  static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
>                 bool remove)
>  {
> -       nvme_rdma_free_qe(ctrl->queues[0].device->dev, 
> &ctrl->async_event_sqe,
> -                       sizeof(struct nvme_command), DMA_TO_DEVICE);
>         nvme_rdma_stop_queue(&ctrl->queues[0]);
>         if (remove) {
>                 blk_cleanup_queue(ctrl->ctrl.admin_q);
> -- 
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme



From jthumshirn at suse.de  Thu Oct 19 01:31:26 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 10:31:26 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171018165258.23212-7-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:47 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-7-hch@lst.de>
Message-ID: <mqdr2tz1qu9.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Thu Oct 19 01:32:18 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 10:32:18 +0200
Subject: [PATCH 05/17] block: don't look at the struct device dev_t in
 disk_devt
In-Reply-To: <20171018165258.23212-6-hch@lst.de> (Christoph Hellwig's message
 of "Wed, 18 Oct 2017 18:52:46 +0200")
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-6-hch@lst.de>
Message-ID: <mqdlgk71qst.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Thu Oct 19 01:40:43 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 10:40:43 +0200
Subject: [PATCH 2/2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20170928153553.13993-3-jsmart2021@gmail.com> (James Smart's
 message of "Thu, 28 Sep 2017 08:35:53 -0700")
References: <20170928153553.13993-1-jsmart2021@gmail.com>
 <20170928153553.13993-3-jsmart2021@gmail.com>
Message-ID: <mqd376fzg1g.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Thu Oct 19 01:49:26 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 10:49:26 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171019084926.GA14227@lst.de>

>  	if (new) {
>  		ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
> -		if (IS_ERR(ctrl->ctrl.admin_tagset))
> +		if (IS_ERR(ctrl->ctrl.admin_tagset)) {
> +			error = PTR_ERR(ctrl->ctrl.admin_tagset);
>  			goto out_free_queue;
> +		}

Can we return an error code instead and just pass in the tag set
to be allocated instead of the admin flag?A  E.g.:

	error = nvme_rdma_alloc_tagset(&ctrl->ctrl, &ctrl->ctrl.admin_tagset);
	if (error)
		goto out_free_queue;


From hch at lst.de  Thu Oct 19 01:51:11 2017
From: hch at lst.de (Christoph Helwig)
Date: Thu, 19 Oct 2017 10:51:11 +0200
Subject: NVMe target log page offset not supported
In-Reply-To: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
Message-ID: <20171019085111.GB14227@lst.de>

On Wed, Oct 18, 2017 at 11:52:04PM +0000, Verkamp, Daniel wrote:
> Unfortunately, I don't have time right now to write and thoroughly test a patch for this, so I thought I'd at least make it known.  It's on my todo list...

Yes, implementing it should be fairly easy, but I'd like to start
wanting test coverage for things like this.

Chaitanya has a really nice test framework, I think it might be a good
fit for that.


From hch at lst.de  Thu Oct 19 01:56:15 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 10:56:15 +0200
Subject: [PATCH] nvme-pci: fix typos in comments
In-Reply-To: <1508334969-16445-1-git-send-email-dn3108@gmail.com>
References: <1508334969-16445-1-git-send-email-dn3108@gmail.com>
Message-ID: <20171019085615.GC14227@lst.de>

Thanks, applied to nvme-4.15.


From hch at infradead.org  Thu Oct 19 01:56:53 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 01:56:53 -0700
Subject: [PATCH v4] nvmet: synchronize sqhd update
In-Reply-To: <20171018213359.3587-1-jsmart2021@gmail.com>
References: <20171018213359.3587-1-jsmart2021@gmail.com>
Message-ID: <20171019085653.GA2622@infradead.org>

On Wed, Oct 18, 2017 at 02:33:59PM -0700, James Smart wrote:
> In testing target io in read write mix, we did indeed get into cases
> where sqhd didn't update properly and slowly missed enough updates to
> shutdown the queue.
> 
> Protect the updating sqhd by using cmpxchg
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>

Applied to nvme-4.14 with a small changelog fixup.


From hch at infradead.org  Thu Oct 19 01:59:26 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 01:59:26 -0700
Subject: [PATCH 0/3] Add BLK_MQ_F_NO_SCHED flag to admin tag set
In-Reply-To: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
References: <1508330306-26844-1-git-send-email-israelr@mellanox.com>
Message-ID: <20171019085926.GA4974@infradead.org>

Thanks Israel,

applied the whole series to nvme-4.15


From hch at infradead.org  Thu Oct 19 02:01:22 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 02:01:22 -0700
Subject: [PATCH v2] nvme/target: Change max_nsid in subsystem due to
 ns_disable if needed
In-Reply-To: <1508323567-8536-1-git-send-email-sagi@grimberg.me>
References: <1508323567-8536-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171019090122.GB4974@infradead.org>

> +	if (list_empty(&subsys->namespaces)) {
> +		subsys->max_nsid = 0;
> +	} else {
> +		struct nvmet_ns *ns =
> +			list_last_entry(&subsys->namespaces, struct nvmet_ns, dev_link);
> +
> +		subsys->max_nsid = ns->nsid;
> +	}

The line is still too long, but I'll fix it up.


From jthumshirn at suse.de  Thu Oct 19 02:04:50 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 11:04:50 +0200
Subject: NVMe target log page offset not supported
In-Reply-To: <20171019085111.GB14227@lst.de> (Christoph Helwig's message of
 "Thu, 19 Oct 2017 10:51:11 +0200")
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
 <20171019085111.GB14227@lst.de>
Message-ID: <mqdh8uv1pa7.fsf@linux-x5ow.site>

Christoph Helwig <hch at lst.de> writes:
> Chaitanya has a really nice test framework, I think it might be a good

Do you (or Chaitanya) have a link to this test framework? You
promised it to me quite some time ago ;-)

Thanks,
        Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Thu Oct 19 02:41:21 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 12:41:21 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <20171019084926.GA14227@lst.de>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
Message-ID: <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>


>>   	if (new) {
>>   		ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
>> -		if (IS_ERR(ctrl->ctrl.admin_tagset))
>> +		if (IS_ERR(ctrl->ctrl.admin_tagset)) {
>> +			error = PTR_ERR(ctrl->ctrl.admin_tagset);
>>   			goto out_free_queue;
>> +		}
> 
> Can we return an error code instead and just pass in the tag set
> to be allocated instead of the admin flag?A  E.g.:
> 
> 	error = nvme_rdma_alloc_tagset(&ctrl->ctrl, &ctrl->ctrl.admin_tagset);
> 	if (error)
> 		goto out_free_queue;

We could, but that would mean that the nvme_ctrl tagset pointer will
need to be updated later (or from the routine itself).

I'm intending this code for nvme-core (and this would be ->alloc_tagset
callout) so ultimately I'd like to move the tagset embedding in struct
nvme_ctrl but given that its a hot-path reference, I suggest we hold it
back until we make struct nvme_ctrl arranged better for the hot path
dereferencing.


From sagi at grimberg.me  Thu Oct 19 02:44:13 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 12:44:13 +0300
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <0561ed58-7104-d8b6-1709-ea181ac493f5@redhat.com>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
 <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
 <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>
 <62eea88b-caa4-5799-3d8f-8d8789879aa8@grimberg.me>
 <0561ed58-7104-d8b6-1709-ea181ac493f5@redhat.com>
Message-ID: <0069d38b-efb7-0d41-ff0f-2087572cb22f@grimberg.me>


> Hi Sagi
> With this patch, no such error log found on host side,


Awsome. that's the culprit...

> but I found there 
> is no nvme0n1 device node even get "nvme nvme0: Successfully 
> reconnected" on host.

That is expected because you did not persist a namespace UUID
which caused the kernel to generate a random one. That confused
the host as it got the same namespace ID with a different UUID.

Can you please set a uuid when you rerun the test?


From Chaitanya.Kulkarni at wdc.com  Thu Oct 19 02:56:20 2017
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 19 Oct 2017 09:56:20 +0000
Subject: NVMe target log page offset not supported
In-Reply-To: <mqdh8uv1pa7.fsf@linux-x5ow.site>
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
 <20171019085111.GB14227@lst.de>,<mqdh8uv1pa7.fsf@linux-x5ow.site>
Message-ID: <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>

Hi Johannes,

I was able to incorporate all the comments received on the first version and added some new testcases.

Please have a look at the following link:-

https://github.com/ChaitanayaKulkarni/nvmftests.

Following is the list of the changes since V1 :-

    1. Addition of new testcases to improve the coverage.
    2. Improved directory structure for code clarity.
    3. Libification of the common code.
    4. Support to use different block devices on the target side.
    5. Centralize test configuration management.
    6. Support to execute File System related operations on the host side.
    7. Adjust the code to incorporate the comments received for the
       first version.
    8. Support for setting different log levels for each component.
    9. Support for quick performance measurements.
    10. Class hierarchy documentation and sequence diagram.
    11. Improved workqueue management for host namesapce to
        run parallel commands.

It will be great if I can get some feedback, I'll be happy to incorporate
the comments and make it more usable.

After completing the comments for this version may be we can tag a release.

Regards,
-Chaitanya







From: Johannes Thumshirn <jthumshirn at suse.de>
Sent: Thursday, October 19, 2017 2:04 AM
To: Christoph Helwig
Cc: Verkamp, Daniel; Sagi Grimberg; linux-nvme at lists.infradead.org; Chaitanya Kulkarni; Hannes Reinecke
Subject: Re: NVMe target log page offset not supported
?    
Christoph Helwig <hch at lst.de> writes:
> Chaitanya has a really nice test framework, I think it might be a good

Do you (or Chaitanya) have a link to this test framework? You
promised it to me quite some time ago ;-)

Thanks,
??????? Johannes
-- 
Johannes Thumshirn????????????????????????????????????????? Storage
jthumshirn at suse.de??????????????????????????????? +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
    

From sagi at grimberg.me  Thu Oct 19 03:01:24 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:01:24 +0300
Subject: NVMe target log page offset not supported
In-Reply-To: <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
 <20171019085111.GB14227@lst.de> <mqdh8uv1pa7.fsf@linux-x5ow.site>
 <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>
Message-ID: <6d0126ad-90bd-11c2-ad4d-21e6fd32c0a7@grimberg.me>

> Hi Johannes,
> 
> I was able to incorporate all the comments received on the first version and added some new testcases.
> 
> Please have a look at the following link:-
> 
> https://github.com/ChaitanayaKulkarni/nvmftests.
> 
> Following is the list of the changes since V1 :-
> 
>      1. Addition of new testcases to improve the coverage.
>      2. Improved directory structure for code clarity.
>      3. Libification of the common code.
>      4. Support to use different block devices on the target side.
>      5. Centralize test configuration management.
>      6. Support to execute File System related operations on the host side.
>      7. Adjust the code to incorporate the comments received for the
>         first version.
>      8. Support for setting different log levels for each component.
>      9. Support for quick performance measurements.
>      10. Class hierarchy documentation and sequence diagram.
>      11. Improved workqueue management for host namesapce to
>          run parallel commands.

Hi Chaitanya,

Would it be possible to post patches so we can effectively provide
feedback?

One thing by a 5 seconds browse, its a shame that you work directly
on top of configfs and not import nvme plugin from nvmetcli repo...


From sagi at grimberg.me  Thu Oct 19 03:02:59 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:02:59 +0300
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct
 device
In-Reply-To: <20171019073741.GA12348@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
 <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
 <20171019073741.GA12348@lst.de>
Message-ID: <827bd4b9-4228-32e4-e8dd-17679f2dfeba@grimberg.me>


>> I don't think the fabrics device does not help us to keep the ctrl
>> allocated until we finish removal.
> 
> All fabrics drivers grab an extra reference during ->create_ctrl,
> which we will drop when releasing the file descriptor that we used
> to create the device.
> 
> By the time we call ->delete_ctrl in nvmf_create_ctrl we must still
> have that reference because we are still in a ->write call and thus
> ->release can't have been called yet.

Thanks for the clarification. Is it worth a preparatory patch to
change that before the ctrl device integration for change log
clarity?


From jthumshirn at suse.de  Thu Oct 19 03:18:08 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 12:18:08 +0200
Subject: NVMe target log page offset not supported
In-Reply-To: <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>
 (Chaitanya Kulkarni's message of "Thu, 19 Oct 2017 09:56:20 +0000")
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
 <20171019085111.GB14227@lst.de> <mqdh8uv1pa7.fsf@linux-x5ow.site>
 <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>
Message-ID: <mqd8tg7bfvj.fsf@linux-x5ow.site>

Chaitanya Kulkarni <Chaitanya.Kulkarni at wdc.com> writes:

> Hi Johannes,
>
> I was able to incorporate all the comments received on the first version and added some new testcases.
>
> Please have a look at the following link:-
>
> https://github.com/ChaitanayaKulkarni/nvmftests.

I'll have a look at this version.

Thanks a lot,
       Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Thu Oct 19 03:18:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 12:18:55 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use
 struct device
In-Reply-To: <827bd4b9-4228-32e4-e8dd-17679f2dfeba@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
 <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
 <20171019073741.GA12348@lst.de>
 <827bd4b9-4228-32e4-e8dd-17679f2dfeba@grimberg.me>
Message-ID: <20171019101855.GA17326@lst.de>

On Thu, Oct 19, 2017 at 01:02:59PM +0300, Sagi Grimberg wrote:
>> By the time we call ->delete_ctrl in nvmf_create_ctrl we must still
>> have that reference because we are still in a ->write call and thus
>> ->release can't have been called yet.
>
> Thanks for the clarification. Is it worth a preparatory patch to
> change that before the ctrl device integration for change log
> clarity?

Well, the ctrl device integration is what enables us to do this.
Before that the ctrl refcount could have reached null by the time
we call the delete_ctrl method.


From hch at lst.de  Thu Oct 19 03:20:22 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 12:20:22 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
Message-ID: <20171019102022.GC17326@lst.de>

>> Can we return an error code instead and just pass in the tag set
>> to be allocated instead of the admin flag?A  E.g.:
>>
>> 	error = nvme_rdma_alloc_tagset(&ctrl->ctrl, &ctrl->ctrl.admin_tagset);
>> 	if (error)
>> 		goto out_free_queue;
>
> We could, but that would mean that the nvme_ctrl tagset pointer will
> need to be updated later (or from the routine itself).

That why I want to pass the address of the tagset pointers, so that
it can be updated.


From hch at infradead.org  Thu Oct 19 03:28:47 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 03:28:47 -0700
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <20171019102847.GA6828@infradead.org>

Keith, Sagi: any comments on this version?  I'd like to get this
work queued up given that we're basically down to nitpicks and future
enhancements for this round.


From sagi at grimberg.me  Thu Oct 19 03:28:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:28:42 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <20171019102022.GC17326@lst.de>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
Message-ID: <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>


>>> to be allocated instead of the admin flag?A  E.g.:
>>>
>>> 	error = nvme_rdma_alloc_tagset(&ctrl->ctrl, &ctrl->ctrl.admin_tagset);
>>> 	if (error)
>>> 		goto out_free_queue;
>>
>> We could, but that would mean that the nvme_ctrl tagset pointer will
>> need to be updated later (or from the routine itself).
> 
> That why I want to pass the address of the tagset pointers, so that
> it can be updated.

But I'm targeting this code to nvme-core which at this point, only has
a pointer to the tagsets (which are embedded in the transport ctrl
structs).


From sagi at grimberg.me  Thu Oct 19 03:31:56 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:31:56 +0300
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <20171019102847.GA6828@infradead.org>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
 <20171019102847.GA6828@infradead.org>
Message-ID: <1022c503-cdb2-e62d-36a9-1932492de812@grimberg.me>


> Keith, Sagi: any comments on this version?  I'd like to get this
> work queued up given that we're basically down to nitpicks and future
> enhancements for this round.

Nothing urgent on my end, I think we're good to go at this point,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Thu Oct 19 03:33:55 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:33:55 +0300
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct
 device
In-Reply-To: <20171019101855.GA17326@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
 <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
 <20171019073741.GA12348@lst.de>
 <827bd4b9-4228-32e4-e8dd-17679f2dfeba@grimberg.me>
 <20171019101855.GA17326@lst.de>
Message-ID: <58a30ee0-f268-2d92-eccd-9d241402a460@grimberg.me>


>>> By the time we call ->delete_ctrl in nvmf_create_ctrl we must still
>>> have that reference because we are still in a ->write call and thus
>>> ->release can't have been called yet.
>>
>> Thanks for the clarification. Is it worth a preparatory patch to
>> change that before the ctrl device integration for change log
>> clarity?
> 
> Well, the ctrl device integration is what enables us to do this.
> Before that the ctrl refcount could have reached null by the time
> we call the delete_ctrl method.

OK, maybe a change log mention then?


From sagi at grimberg.me  Thu Oct 19 03:35:58 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:35:58 +0300
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171018165258.23212-4-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-4-hch@lst.de>
Message-ID: <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>


> +/**
> + * direct_make_request - hand a buffer directly to its device driver for I/O
> + * @bio:  The bio describing the location in memory and on the device.
> + *
> + * This function behaves like generic_make_request(), but does not protect
> + * against recursion.  Must only be used if the called driver is known
> + * to not call generic_make_request (or direct_make_request) again from
> + * its make_request function.  (Calling direct_make_request again from
> + * a workqueue is perfectly fine as that doesn't recurse).

Question: are the double spaces on purpose?


From Chaitanya.Kulkarni at wdc.com  Thu Oct 19 03:38:25 2017
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 19 Oct 2017 10:38:25 +0000
Subject: NVMe target log page offset not supported
In-Reply-To: <6d0126ad-90bd-11c2-ad4d-21e6fd32c0a7@grimberg.me>
References: <A5F28D4A728A7E41839CDC5C3B5A01E887295CEF@FMSMSX103.amr.corp.intel.com>
 <20171019085111.GB14227@lst.de> <mqdh8uv1pa7.fsf@linux-x5ow.site>
 <MWHPR04MB1008D60CB5469DD12F6896BE86420@MWHPR04MB1008.namprd04.prod.outlook.com>,
 <6d0126ad-90bd-11c2-ad4d-21e6fd32c0a7@grimberg.me>
Message-ID: <MWHPR04MB10088DF73F1E6CEB0F07F09F86420@MWHPR04MB1008.namprd04.prod.outlook.com>

Sure, I'll post a patch series for the latest version.

With the new structure of the code, it should be easy to add nvme plug-in from the nvmetcli.

Prior to that what I'm looking for in the comments:-

1. What are new file system types we should add in the utils? 
2. Is threading model on the host side needs to be replicated on the target side?
3. Right now we have a single thread for each namespace and associated work queue to execute the command on the namespace for the host, similar to namespace do we really need a separate thread for each controller also on the host side? What about target side controller.
4. Any specific fio workloads need to be added for basic testing?

Any other comments.

Regards,
-Chaitanya
  
From: Sagi Grimberg <sagi at grimberg.me>
Sent: Thursday, October 19, 2017 3:01:24 AM
To: Chaitanya Kulkarni; Johannes Thumshirn
Cc: Verkamp, Daniel; linux-nvme at lists.infradead.org; Hannes Reinecke; Christoph Hellwig
Subject: Re: NVMe target log page offset not supported
?   
> Hi Johannes,
> 
> I was able to incorporate all the comments received on the first version and added some new testcases.
> 
> Please have a look at the following link:-
> 
> https://github.com/ChaitanayaKulkarni/nvmftests.

 https://avatars3.githubusercontent.com/u/24584550?s=400&v=4 

ChaitanayaKulkarni/nvmftests
github.com
Contribute to nvmftests development by creating an account on GitHub.

> 
> Following is the list of the changes since V1 :-
> 
>????? 1. Addition of new testcases to improve the coverage.
>????? 2. Improved directory structure for code clarity.
>????? 3. Libification of the common code.
>????? 4. Support to use different block devices on the target side.
>????? 5. Centralize test configuration management.
>????? 6. Support to execute File System related operations on the host side.
>????? 7. Adjust the code to incorporate the comments received for the
>???????? first version.
>????? 8. Support for setting different log levels for each component.
>????? 9. Support for quick performance measurements.
>????? 10. Class hierarchy documentation and sequence diagram.
>????? 11. Improved workqueue management for host namesapce to
>????????? run parallel commands.

Hi Chaitanya,

Would it be possible to post patches so we can effectively provide
feedback?

One thing by a 5 seconds browse, its a shame that you work directly
on top of configfs and not import nvme plugin from nvmetcli repo...
    

From sagi at grimberg.me  Thu Oct 19 04:00:57 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 14:00:57 +0300
Subject: [PATCH 4.14-rc fix-regression] nvme-rdma: Fix possible double free in
 reconnect flow
Message-ID: <1508410857-13678-1-git-send-email-sagi@grimberg.me>

The fact that we free the async event buffer in
nvme_rdma_destroy_admin_queue can cause us to free it
more than once because this happens in every reconnect
attempt (we rely on the queue state flags DELETING
to avoid this for other resources).

A more complete fix is to not destroy the admin/io queues
unconditionally on every reconnect attempt, but its a bit
more extensive and will go in the next release.

Reported-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
Note that this might cause a minor conflict with nvme-4.15
patches, but it will be easy enough to resolve it.

 drivers/nvme/host/rdma.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..ec22da478ba3 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -571,6 +571,11 @@ static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
 	if (test_and_set_bit(NVME_RDMA_Q_DELETING, &queue->flags))
 		return;
 
+	if(nvme_rdma_queue_idx(queue) == 0)
+		nvme_rdma_free_qe(queue->device->dev,
+			&queue->ctrl->async_event_sqe,
+			sizeof(struct nvme_command), DMA_TO_DEVICE);
+
 	nvme_rdma_destroy_queue_ib(queue);
 	rdma_destroy_id(queue->cm_id);
 }
@@ -739,8 +744,6 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
 static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
 		bool remove)
 {
-	nvme_rdma_free_qe(ctrl->queues[0].device->dev, &ctrl->async_event_sqe,
-			sizeof(struct nvme_command), DMA_TO_DEVICE);
 	nvme_rdma_stop_queue(&ctrl->queues[0]);
 	if (remove) {
 		blk_cleanup_queue(ctrl->ctrl.admin_q);
-- 
2.7.4



From sagi at grimberg.me  Thu Oct 19 04:06:07 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 14:06:07 +0300
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171018165258.23212-16-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-16-hch@lst.de>
Message-ID: <3c68d0c8-a063-44f9-a76b-0970c9e4ad5d@grimberg.me>

Christoph,

>   static void nvme_free_ns(struct kref *kref)
>   {
>   	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
>   
> +	if (ns->head)
> +		nvme_put_ns_head(ns->head);
> +

When can we not have a ns-head set?

AFAICT, if nvme_alloc_ns succeeded, we have it set don't we?


From jthumshirn at suse.de  Thu Oct 19 04:06:40 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 13:06:40 +0200
Subject: [PATCH 4.14-rc fix-regression] nvme-rdma: Fix possible double
 free in reconnect flow
In-Reply-To: <1508410857-13678-1-git-send-email-sagi@grimberg.me> (Sagi
 Grimberg's message of "Thu, 19 Oct 2017 14:00:57 +0300")
References: <1508410857-13678-1-git-send-email-sagi@grimberg.me>
Message-ID: <mqd376fbdmn.fsf@linux-x5ow.site>

Sagi Grimberg <sagi at grimberg.me> writes:

> The fact that we free the async event buffer in
> nvme_rdma_destroy_admin_queue can cause us to free it
> more than once because this happens in every reconnect
> attempt (we rely on the queue state flags DELETING
> to avoid this for other resources).

Do you appen to know which commit introduced the regression? A fixes
line/tag would be really nice here.

Thanks,
        Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Thu Oct 19 04:07:18 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 13:07:18 +0200
Subject: [PATCH 4.14-rc fix-regression] nvme-rdma: Fix possible double
 free in reconnect flow
In-Reply-To: <1508410857-13678-1-git-send-email-sagi@grimberg.me> (Sagi
 Grimberg's message of "Thu, 19 Oct 2017 14:00:57 +0300")
References: <1508410857-13678-1-git-send-email-sagi@grimberg.me>
Message-ID: <mqd1slzbdll.fsf@linux-x5ow.site>

Sagi Grimberg <sagi at grimberg.me> writes:

> The fact that we free the async event buffer in
> nvme_rdma_destroy_admin_queue can cause us to free it
> more than once because this happens in every reconnect
> attempt (we rely on the queue state flags DELETING
> to avoid this for other resources).

Do you appen to know which commit introduced the regression? A fixes
line/tag would be really nice here.

Thanks,
        Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From yi.zhang at redhat.com  Thu Oct 19 04:13:43 2017
From: yi.zhang at redhat.com (Yi Zhang)
Date: Thu, 19 Oct 2017 19:13:43 +0800
Subject: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)
In-Reply-To: <0069d38b-efb7-0d41-ff0f-2087572cb22f@grimberg.me>
References: <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
 <1215229914.8516804.1505998051674.JavaMail.zimbra@redhat.com>
 <20170921144421.GA15285@infradead.org>
 <47493aa0-4cad-721b-4ea2-c3b2293340aa@grimberg.me>
 <e8aa9f2e-14b1-23f0-f672-07a913842c9d@redhat.com>
 <20170924103426.GB25094@mtr-leonro.local>
 <4d0e27f2-99e4-ffc7-18a5-e5757c27dad4@redhat.com>
 <4faccebf-7fcf-58b9-1605-82ee9acc652b@grimberg.me>
 <e44f4640-44b2-4692-ca57-08f9ebf4da2c@redhat.com>
 <729c512f-55ff-25e4-6dd9-8b4dcc31bb8d@grimberg.me>
 <ff355fcf-5089-f297-d3fa-9b86d133236b@redhat.com>
 <6d639dd4-9366-2ecf-b49e-751c7a167528@grimberg.me>
 <64135330-a62d-87b8-c805-6b52fd40bf81@redhat.com>
 <62eea88b-caa4-5799-3d8f-8d8789879aa8@grimberg.me>
 <0561ed58-7104-d8b6-1709-ea181ac493f5@redhat.com>
 <0069d38b-efb7-0d41-ff0f-2087572cb22f@grimberg.me>
Message-ID: <212f5da2-b0e2-23aa-d517-7512d656dec8@redhat.com>



On 10/19/2017 05:44 PM, Sagi Grimberg wrote:
>
>> Hi Sagi
>> With this patch, no such error log found on host side,
>
>
> Awsome. that's the culprit...
>
>> but I found there is no nvme0n1 device node even get "nvme nvme0: 
>> Successfully reconnected" on host.
>
> That is expected because you did not persist a namespace UUID
> which caused the kernel to generate a random one. That confused
> the host as it got the same namespace ID with a different UUID.
>
> Can you please set a uuid when you rerun the test?
>
I tried add uuid filed on rdma.json, it works well now.

[root at rdma-virt-00 ~]$ cat /etc/rdma.json
{
   "hosts": [
     {
       "nqn": "hostnqn"
     }
   ],
   "ports": [
     {
       "addr": {
         "adrfam": "ipv4",
         "traddr": "172.31.0.90",
         "treq": "not specified",
         "trsvcid": "4420",
         "trtype": "rdma"
       },
       "portid": 2,
       "referrals": [],
       "subsystems": [
         "testnqn"
       ]
     }
   ],
   "subsystems": [
     {
       "allowed_hosts": [],
       "attr": {
         "allow_any_host": "1"
       },
       "namespaces": [
         {
           "device": {
             "nguid": "ef90689c-6c46-d44c-89c1-4067801309a8",
             "path": "/dev/nullb0",
             "uuid": "00000000-0000-0000-0000-000000000001"
           },
           "enable": 1,
           "nsid": 1
         }
       ],
       "nqn": "testnqn"
     }
   ]
}

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme



From sagi at grimberg.me  Thu Oct 19 03:36:50 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 13:36:50 +0300
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-4-hch@lst.de>
 <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>
Message-ID: <b4a2622d-a228-a6d8-769d-5b684ceff922@grimberg.me>


>> +/**
>> + * direct_make_request - hand a buffer directly to its device driver 
>> for I/O
>> + * @bio:? The bio describing the location in memory and on the device.
>> + *
>> + * This function behaves like generic_make_request(), but does not 
>> protect
>> + * against recursion.? Must only be used if the called driver is known
>> + * to not call generic_make_request (or direct_make_request) again from
>> + * its make_request function.? (Calling direct_make_request again from
>> + * a workqueue is perfectly fine as that doesn't recurse).
> 
> Question: are the double spaces on purpose?

Other than that,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From hare at suse.de  Thu Oct 19 05:45:03 2017
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 19 Oct 2017 14:45:03 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171018165258.23212-7-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-7-hch@lst.de>
Message-ID: <1d2fe3fa-564f-c056-bbc5-b80282bee912@suse.de>

On 10/18/2017 06:52 PM, Christoph Hellwig wrote:
> With this flag a driver can create a gendisk that can be used for I/O
> submission inside the kernel, but which is not registered as user
> facing block device.  This will be useful for the NVMe multipath
> implementation.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  block/genhd.c         | 53 ++++++++++++++++++++++++++++++++++-----------------
>  include/linux/genhd.h |  1 +
>  2 files changed, 36 insertions(+), 18 deletions(-)
> 
To quote:

On Fri, Oct 06, 2017 at 09:13 AM, Christoph Hellwig wrote:
> On Fri, Oct 06, 2017 at 10:39:50AM +1100, NeilBrown wrote:
[ .. ]
>>
>> There is some precedent for hiding things from /proc/partitions.
>> removable devices like CDROMs are hidden, and you can easily hide
>> individual devices by setting GENHD_FL_SUPPRESS_PARTITION_INFO.
>> We might be able to get that through.  It is certainly worth writing
>> a patch and letting people experiment with it.
>
> No way in hell this would get through.  Preemptive NAK right here.
>
> That whole idea is just amazingly stupid, and no one has even
> explained a reason for it.

I wonder what has changed ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From sagi at grimberg.me  Thu Oct 19 06:00:30 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 16:00:30 +0300
Subject: [PATCH v2 4.14-rc fix-regression] nvme-rdma: Fix possible double free
 in reconnect flow
Message-ID: <1508418030-23031-1-git-send-email-sagi@grimberg.me>

The fact that we free the async event buffer in
nvme_rdma_destroy_admin_queue can cause us to free it
more than once because this happens in every reconnect
attempt since commit 31fdf1840170. we rely on the queue
state flags DELETING to avoid this for other resources.

A more complete fix is to not destroy the admin/io queues
unconditionally on every reconnect attempt, but its a bit
more extensive and will go in the next release.

Fixes: 31fdf1840170 ("nvme-rdma: reuse configure/destroy_admin_queue")
Reported-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..ec22da478ba3 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -571,6 +571,11 @@ static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
 	if (test_and_set_bit(NVME_RDMA_Q_DELETING, &queue->flags))
 		return;
 
+	if(nvme_rdma_queue_idx(queue) == 0)
+		nvme_rdma_free_qe(queue->device->dev,
+			&queue->ctrl->async_event_sqe,
+			sizeof(struct nvme_command), DMA_TO_DEVICE);
+
 	nvme_rdma_destroy_queue_ib(queue);
 	rdma_destroy_id(queue->cm_id);
 }
@@ -739,8 +744,6 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
 static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
 		bool remove)
 {
-	nvme_rdma_free_qe(ctrl->queues[0].device->dev, &ctrl->async_event_sqe,
-			sizeof(struct nvme_command), DMA_TO_DEVICE);
 	nvme_rdma_stop_queue(&ctrl->queues[0]);
 	if (remove) {
 		blk_cleanup_queue(ctrl->ctrl.admin_q);
-- 
2.7.4



From hch at lst.de  Thu Oct 19 06:15:35 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:15:35 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <1d2fe3fa-564f-c056-bbc5-b80282bee912@suse.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-7-hch@lst.de>
 <1d2fe3fa-564f-c056-bbc5-b80282bee912@suse.de>
Message-ID: <20171019131535.GB20645@lst.de>

On Thu, Oct 19, 2017 at 02:45:03PM +0200, Hannes Reinecke wrote:
> > No way in hell this would get through.  Preemptive NAK right here.
> >
> > That whole idea is just amazingly stupid, and no one has even
> > explained a reason for it.
> 
> I wonder what has changed ...

The nvme-controller devices aren't accessible to anyone outside the
nvme driver itself, so they aren't just hidden, as far as the rest
of the kernel is concerned they don't exist.  They just allow reusing
block layer code in the nvme driver instead of reimplementing it.


From jthumshirn at suse.de  Thu Oct 19 06:32:52 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Thu, 19 Oct 2017 15:32:52 +0200
Subject: [PATCH v2 4.14-rc fix-regression] nvme-rdma: Fix possible double
 free in reconnect flow
In-Reply-To: <1508418030-23031-1-git-send-email-sagi@grimberg.me> (Sagi
 Grimberg's message of "Thu, 19 Oct 2017 16:00:30 +0300")
References: <1508418030-23031-1-git-send-email-sagi@grimberg.me>
Message-ID: <mqdr2tz8dq3.fsf@linux-x5ow.site>

Thanks a lot,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Thu Oct 19 06:51:51 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:51:51 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <3c68d0c8-a063-44f9-a76b-0970c9e4ad5d@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-16-hch@lst.de>
 <3c68d0c8-a063-44f9-a76b-0970c9e4ad5d@grimberg.me>
Message-ID: <20171019135151.GB21452@lst.de>

On Thu, Oct 19, 2017 at 02:06:07PM +0300, Sagi Grimberg wrote:
> Christoph,
>
>>   static void nvme_free_ns(struct kref *kref)
>>   {
>>   	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
>>   +	if (ns->head)
>> +		nvme_put_ns_head(ns->head);
>> +
>
> When can we not have a ns-head set?
>
> AFAICT, if nvme_alloc_ns succeeded, we have it set don't we?

Yes, I don't think we should need it.  Let me verify that for the
next round.


From hch at lst.de  Thu Oct 19 06:52:32 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:52:32 +0200
Subject: [PATCH 4.14-rc fix-regression] nvme-rdma: Fix possible double
 free in reconnect flow
In-Reply-To: <1508410857-13678-1-git-send-email-sagi@grimberg.me>
References: <1508410857-13678-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171019135232.GC21452@lst.de>

> +	if(nvme_rdma_queue_idx(queue) == 0)

missing space before the opening brace, but I can fix that up.



From hch at lst.de  Thu Oct 19 06:54:30 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:54:30 +0200
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-4-hch@lst.de>
 <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>
Message-ID: <20171019135430.GD21452@lst.de>

On Thu, Oct 19, 2017 at 01:35:58PM +0300, Sagi Grimberg wrote:
>
>> +/**
>> + * direct_make_request - hand a buffer directly to its device driver for I/O
>> + * @bio:  The bio describing the location in memory and on the device.
>> + *
>> + * This function behaves like generic_make_request(), but does not protect
>> + * against recursion.  Must only be used if the called driver is known
>> + * to not call generic_make_request (or direct_make_request) again from
>> + * its make_request function.  (Calling direct_make_request again from
>> + * a workqueue is perfectly fine as that doesn't recurse).
>
> Question: are the double spaces on purpose?

Yes.  It's a very common style, and especially useful with monospace fonts.
https://en.wikipedia.org/wiki/Sentence_spacing for reference.


From hch at lst.de  Thu Oct 19 06:54:48 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:54:48 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use
 struct device
In-Reply-To: <58a30ee0-f268-2d92-eccd-9d241402a460@grimberg.me>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-11-hch@lst.de>
 <b4e85bf7-f0ae-4ea9-2caa-3ba3645f88ea@grimberg.me>
 <20171019072053.GA12168@lst.de>
 <05edcff6-4c56-39d9-83c8-e3eae03b824d@grimberg.me>
 <20171019073741.GA12348@lst.de>
 <827bd4b9-4228-32e4-e8dd-17679f2dfeba@grimberg.me>
 <20171019101855.GA17326@lst.de>
 <58a30ee0-f268-2d92-eccd-9d241402a460@grimberg.me>
Message-ID: <20171019135448.GE21452@lst.de>

On Thu, Oct 19, 2017 at 01:33:55PM +0300, Sagi Grimberg wrote:
>> Well, the ctrl device integration is what enables us to do this.
>> Before that the ctrl refcount could have reached null by the time
>> we call the delete_ctrl method.
>
> OK, maybe a change log mention then?

I thought I did that, but I can be more detailed.


From hch at lst.de  Thu Oct 19 06:55:22 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 15:55:22 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
 <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
Message-ID: <20171019135522.GF21452@lst.de>

On Thu, Oct 19, 2017 at 01:28:42PM +0300, Sagi Grimberg wrote:
>>> We could, but that would mean that the nvme_ctrl tagset pointer will
>>> need to be updated later (or from the routine itself).
>>
>> That why I want to pass the address of the tagset pointers, so that
>> it can be updated.
>
> But I'm targeting this code to nvme-core which at this point, only has
> a pointer to the tagsets (which are embedded in the transport ctrl
> structs).

But with a a double indirection you _can_ update the pointers.


From sagi at grimberg.me  Thu Oct 19 07:42:46 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 17:42:46 +0300
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171019135430.GD21452@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-4-hch@lst.de>
 <d8605553-ef38-4aa2-1513-39ae49de3856@grimberg.me>
 <20171019135430.GD21452@lst.de>
Message-ID: <254bc6d0-31ea-4719-baf5-2d847356db97@grimberg.me>


>>> +/**
>>> + * direct_make_request - hand a buffer directly to its device driver for I/O
>>> + * @bio:  The bio describing the location in memory and on the device.
>>> + *
>>> + * This function behaves like generic_make_request(), but does not protect
>>> + * against recursion.  Must only be used if the called driver is known
>>> + * to not call generic_make_request (or direct_make_request) again from
>>> + * its make_request function.  (Calling direct_make_request again from
>>> + * a workqueue is perfectly fine as that doesn't recurse).
>>
>> Question: are the double spaces on purpose?
> 
> Yes.  It's a very common style, and especially useful with monospace fonts.
> https://en.wikipedia.org/wiki/Sentence_spacing for reference.
> 

Thanks.


From sagi at grimberg.me  Thu Oct 19 07:44:57 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 17:44:57 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <20171019135522.GF21452@lst.de>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
 <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
 <20171019135522.GF21452@lst.de>
Message-ID: <d5177951-d869-786d-77ad-679929daf834@grimberg.me>


>>>> We could, but that would mean that the nvme_ctrl tagset pointer will
>>>> need to be updated later (or from the routine itself).
>>>
>>> That why I want to pass the address of the tagset pointers, so that
>>> it can be updated.
>>
>> But I'm targeting this code to nvme-core which at this point, only has
>> a pointer to the tagsets (which are embedded in the transport ctrl
>> structs).
> 
> But with a a double indirection you _can_ update the pointers.

Oh, double indirection... I confused the ctrl structures in your
suggestion...

Sure I can do that.

Is that still fine for for 4.14-rc?


From hch at lst.de  Thu Oct 19 07:49:26 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 16:49:26 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <d5177951-d869-786d-77ad-679929daf834@grimberg.me>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
 <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
 <20171019135522.GF21452@lst.de>
 <d5177951-d869-786d-77ad-679929daf834@grimberg.me>
Message-ID: <20171019144925.GA23650@lst.de>

On Thu, Oct 19, 2017 at 05:44:57PM +0300, Sagi Grimberg wrote:
> Oh, double indirection... I confused the ctrl structures in your
> suggestion...

Actually, let's skip the idea - I just noticed nvme_rdma_alloc_tagset does
way too many different things for the admin vs io taget.

I'm fine with your original version, but once we refactor things in this
area I have a few ideas how to improve things.


From sagi at grimberg.me  Thu Oct 19 07:58:16 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 17:58:16 +0300
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <20171019144925.GA23650@lst.de>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
 <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
 <20171019135522.GF21452@lst.de>
 <d5177951-d869-786d-77ad-679929daf834@grimberg.me>
 <20171019144925.GA23650@lst.de>
Message-ID: <94390d33-dc54-b8ae-ed9a-ba49c4f71ae0@grimberg.me>


> Actually, let's skip the idea - I just noticed nvme_rdma_alloc_tagset does
> way too many different things for the admin vs io taget.
> 
> I'm fine with your original version, but once we refactor things in this
> area I have a few ideas how to improve things.

I have no doubt you do...


From hch at lst.de  Thu Oct 19 08:07:50 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 17:07:50 +0200
Subject: [PATCH v2 4.14-rc fix-regression] nvme-rdma: Fix possible
 double free in reconnect flow
In-Reply-To: <1508418030-23031-1-git-send-email-sagi@grimberg.me>
References: <1508418030-23031-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171019150750.GA24204@lst.de>

Applied to nvme-4.14 after fixing this:

> +	if(nvme_rdma_queue_idx(queue) == 0)

whitespace issue and adding some curly braces.


From sagi at grimberg.me  Thu Oct 19 08:10:53 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 18:10:53 +0300
Subject: [PATCH v2 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
Message-ID: <1508425853-31998-1-git-send-email-sagi@grimberg.me>

We should make sure to escelate allocation failures to
prevent a use-after-free in nvmf_create_ctrl.

Fixes: b28a308ee777 ("nvme-rdma: move tagset allocation to
a dedicated routine")
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
Chnages from v1:
- fixed changelog

 drivers/nvme/host/rdma.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index a5577e06a620..1f01744d859f 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -766,8 +766,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 
 	if (new) {
 		ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
-		if (IS_ERR(ctrl->ctrl.admin_tagset))
+		if (IS_ERR(ctrl->ctrl.admin_tagset)) {
+			error = PTR_ERR(ctrl->ctrl.admin_tagset);
 			goto out_free_queue;
+		}
 
 		ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set);
 		if (IS_ERR(ctrl->ctrl.admin_q)) {
@@ -846,8 +848,10 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
 	if (new) {
 		ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, false);
-		if (IS_ERR(ctrl->ctrl.tagset))
+		if (IS_ERR(ctrl->ctrl.tagset)) {
+			ret = PTR_ERR(ctrl->ctrl.tagset);
 			goto out_free_io_queues;
+		}
 
 		ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set);
 		if (IS_ERR(ctrl->ctrl.connect_q)) {
-- 
2.7.4



From hch at lst.de  Thu Oct 19 08:12:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 17:12:55 +0200
Subject: [PATCH 07/17] nvme: use ida_simple_{get,remove} for the
 controller instance
In-Reply-To: <20171019151426.GC2768@localhost.localdomain>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-8-hch@lst.de>
 <20171019151426.GC2768@localhost.localdomain>
Message-ID: <20171019151255.GA24354@lst.de>

On Thu, Oct 19, 2017 at 09:14:27AM -0600, Keith Busch wrote:
> This is a good cleanup, and I'd support this patch going in ahead of
> this series on its own if you want to apply to 4.15 immediately.

Ok, will do.


From hch at lst.de  Thu Oct 19 08:04:07 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 17:04:07 +0200
Subject: [PATCH 4.14-rc] nvme-rdma: Fix error status return in tagset
 allocation failure
In-Reply-To: <94390d33-dc54-b8ae-ed9a-ba49c4f71ae0@grimberg.me>
References: <1508396289-12237-1-git-send-email-sagi@grimberg.me>
 <20171019084926.GA14227@lst.de>
 <9604c4ac-0a0f-a029-1894-051e6de50072@grimberg.me>
 <20171019102022.GC17326@lst.de>
 <9cb7f448-f4a3-e3e6-e906-749a56daa943@grimberg.me>
 <20171019135522.GF21452@lst.de>
 <d5177951-d869-786d-77ad-679929daf834@grimberg.me>
 <20171019144925.GA23650@lst.de>
 <94390d33-dc54-b8ae-ed9a-ba49c4f71ae0@grimberg.me>
Message-ID: <20171019150407.GA24110@lst.de>

On Thu, Oct 19, 2017 at 05:58:16PM +0300, Sagi Grimberg wrote:
>
>> Actually, let's skip the idea - I just noticed nvme_rdma_alloc_tagset does
>> way too many different things for the admin vs io taget.
>>
>> I'm fine with your original version, but once we refactor things in this
>> area I have a few ideas how to improve things.
>
> I have no doubt you do...

In the meantime can you write a more detailed changelog?  For a late
rc fix I'd really like to have a better description especially with
Linus hating us block and nvme people at the moment :)


From hch at lst.de  Thu Oct 19 08:18:02 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 17:18:02 +0200
Subject: [PATCH v2 4.14-rc] nvme-rdma: Fix error status return in
 tagset allocation failure
In-Reply-To: <1508425853-31998-1-git-send-email-sagi@grimberg.me>
References: <1508425853-31998-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171019151802.GA24556@lst.de>

Thanks,

applied to nvme-4.14.


From sagi at grimberg.me  Thu Oct 19 08:19:41 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 18:19:41 +0300
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <20171018201713.GA1971@localhost.localdomain>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171018150859.GD32530@localhost.localdomain>
 <4627b34e-449a-cd93-ac70-a628d558cac3@grimberg.me>
 <20171018201713.GA1971@localhost.localdomain>
Message-ID: <9942d9a3-89b7-84ee-1579-ab8e00b71e6a@grimberg.me>


> Hm, I'm not hitting duplicate name issues anymore. I can't be bothered
> to bisect when that started working, but I think the dev_t lifetime
> fixes has something to do with it.
> 
> Anyway, simpler is certainly preferred. This is looking okay so far,
> and I'll try a few more things to see if I can break it.

Thanks Keith,

Would be good to get your review/tested tag if it survives your tests.


From hch at infradead.org  Thu Oct 19 08:20:24 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 08:20:24 -0700
Subject: [PATCH 1/3] nvme: add duplicate_connect option
In-Reply-To: <20171011003512.5946-2-jsmart2021@gmail.com>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
 <20171011003512.5946-2-jsmart2021@gmail.com>
Message-ID: <20171019152024.GA4239@infradead.org>

> +	{ NVMF_OPT_DUP_CONNECT,		"duplicate_connect=%d"	},

This is a boolean parameter, so please drop the "=%d here".

> +		case NVMF_OPT_DUP_CONNECT:
> +			if (match_int(args, &token)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (token == 0)
> +				opts->duplicate_connect = false;
> +			else if (token == 1)
> +				opts->duplicate_connect = true;
> +			else {
> +				pr_err("Invalid duplicate_connect %d\n", token);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			break;

And replace all this code by:

	case NVMF_OPT_DUP_CONNECT:
		opts->duplicate_connect = true;
		break;

That being said I'd be tempted to keep the existing behavior as
the default, and make noduplicates the option.


From hch at infradead.org  Thu Oct 19 08:23:17 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 19 Oct 2017 08:23:17 -0700
Subject: [PATCH 2/3] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171011003512.5946-3-jsmart2021@gmail.com>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
 <20171011003512.5946-3-jsmart2021@gmail.com>
Message-ID: <20171019152317.GB4239@infradead.org>

> +static inline bool
> +__nvme_fc_options_match(struct nvmf_ctrl_options *opts,
> +			struct nvme_fc_ctrl *ctrl)
> +{
> +	if (strcmp(opts->subsysnqn, ctrl->ctrl.opts->subsysnqn) ||
> +	    strcmp(opts->host->nqn, ctrl->ctrl.opts->host->nqn) ||
> +	    memcmp(&opts->host->id, &ctrl->ctrl.opts->host->id,
> +				sizeof(uuid_t)))
> +		return false;

This just checks generic options and isn't FC specific.  As a properly
named helper it might make sense in common code.


From sagi at grimberg.me  Thu Oct 19 08:24:26 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 18:24:26 +0300
Subject: [PATCH 17/17] nvme: also expose the namespace identification
 sysfs files for mpath nodes
In-Reply-To: <20171018165258.23212-18-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-18-hch@lst.de>
Message-ID: <44b2f237-3c32-38f9-13d0-9ea70e1166e2@grimberg.me>

I like the fact that we're expose only the device
nodes.

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Thu Oct 19 08:31:38 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 19 Oct 2017 18:31:38 +0300
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <20171019152910.GA24808@lst.de>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171019152910.GA24808@lst.de>
Message-ID: <0a081ffc-5a3d-6b52-7257-18e39034ce93@grimberg.me>


>> From: Roy Shterman <roys at lightbitslabs.com>
>>
>> Its strange to have some of the cleanup for nvme_init_ctrl in
>> nvme_free_ctrl. This cause us to have some magic ctrl refcount
>> handling in controller initialization error paths.
>>
>> Make the init/uninit symmetric so we can have a sane and
>> maintainable error sequences.
> 
> This totally conflicts with my refcount / device rework in the
> mpath patchset..

I know...

Do you want me to rebase on top of your next round of patches?


From hch at lst.de  Thu Oct 19 08:29:10 2017
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 19 Oct 2017 17:29:10 +0200
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl
 setup/teardown symmetric
In-Reply-To: <1508324864-23087-2-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
Message-ID: <20171019152910.GA24808@lst.de>

On Wed, Oct 18, 2017 at 02:07:40PM +0300, Sagi Grimberg wrote:
> From: Roy Shterman <roys at lightbitslabs.com>
> 
> Its strange to have some of the cleanup for nvme_init_ctrl in
> nvme_free_ctrl. This cause us to have some magic ctrl refcount
> handling in controller initialization error paths.
> 
> Make the init/uninit symmetric so we can have a sane and
> maintainable error sequences.

This totally conflicts with my refcount / device rework in the
mpath patchset..


From keith.busch at intel.com  Thu Oct 19 08:48:22 2017
From: keith.busch at intel.com (Keith Busch)
Date: Thu, 19 Oct 2017 09:48:22 -0600
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <20171019102847.GA6828@infradead.org>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
 <20171019102847.GA6828@infradead.org>
Message-ID: <20171019154821.GD2768@localhost.localdomain>

Yeah, I don't have any major concerns. I was hung up on the
tagset.cmd_size being smaller when SGL's are used, but it took me a
minute to determine that's okay as we were overallocating it before.

The criteria used to select sgl usage makes sense, so good to go on my
end.

Reviewed-by: Keith Busch <keith.busch at intel.com>


From keith.busch at intel.com  Thu Oct 19 08:14:27 2017
From: keith.busch at intel.com (Keith Busch)
Date: Thu, 19 Oct 2017 09:14:27 -0600
Subject: [PATCH 07/17] nvme: use ida_simple_{get,remove} for the
 controller instance
In-Reply-To: <20171018165258.23212-8-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-8-hch@lst.de>
Message-ID: <20171019151426.GC2768@localhost.localdomain>

This is a good cleanup, and I'd support this patch going in ahead of
this series on its own if you want to apply to 4.15 immediately.

Reviewed-by: Keith Busch <keith.busch at intel.com>


From Darren.Trapp at cavium.com  Thu Oct 19 08:46:04 2017
From: Darren.Trapp at cavium.com (Trapp, Darren)
Date: Thu, 19 Oct 2017 15:46:04 +0000
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20171017233248.6769-6-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-6-jsmart2021@gmail.com>
Message-ID: <32B40236-CB8E-461E-B765-AEF544AB981A@cavium.com>

It seems like we need a call back to the LLD to let him know that the dev_loss_tmo has been canceled upon device coming back.

The LLD has called the unregister_remoteport in the transport and done a wait_for_completion.  If the device re-appears before dev_loss_tmo has been reached, the transport cancels the del_work.  But, the LLD still has the completion event pending.  

Something like this:

		spin_unlock_irqrestore(&nvme_fc_lock, flags);
   
-  		cancel_delayed_work_sync(&rport->dev_loss_work);
+		if (cancel_delayed_work_sync(&rport->dev_loss_work))
+			if (lport->ops->remote_delete_cancel)
+				lport->ops->remote_delete_cancel(&rport->remoteport);
   
    		spin_lock_irqsave(&rport->lock, flags);
    

The LLD can complete its task and do whatever else it needs to do knowing that the unregister task is dead.  Seems like a clean handoff between the 2.


On 10/17/17, 4:34 PM, "Linux-nvme on behalf of James Smart" <linux-nvme-bounces at lists.infradead.org on behalf of jsmart2021 at gmail.com> wrote:

    This patch adds the dev_loss_tmo functionality to the transport.
    
    When a remoteport is unregistered (connectivity lost), the following
    actions are taken:
    - the remoteport is marked DELETED
    - a dev_loss_tmo timer is started for the remoteport
    - all controllers on the remoteport are reset.
    
    After a controller resets, it will stall in a RECONNECTING state
    waiting for one of the following:
    - the controller will continue to attempt reconnect per max_retries
      and reconnect_delay. As no remoteport connectivity, the reconnect
      attempt will immediately fail. if max reconnect attempts are
      reached (e.g. ctrl_loss_tmo reached) the controller is deleted.
    - the remoteport is re-registered prior to dev_loss_tmo expiring.
      The resume of the remoteport will immediately attempt to reconnect
      each of its suspended controllers.
    - the remoteport's dev_loss_tmo expires, causing all of its
      controllers to be deleted.
    
    Signed-off-by: James Smart <james.smart at broadcom.com>
    
    v3:
      Reworked so dev_loss_tmo specific to the remoteport. Revised
      so connectivity loss resets controllers, and connectivity gain
      schedules reconnect.
    ---
     drivers/nvme/host/fc.c | 289 +++++++++++++++++++++++++++++++++++++++++++------
     1 file changed, 257 insertions(+), 32 deletions(-)
    
    diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
    index 61da2f92f71a..3628790371d7 100644
    --- a/drivers/nvme/host/fc.c
    +++ b/drivers/nvme/host/fc.c
    @@ -138,6 +138,7 @@ struct nvme_fc_rport {
     	struct nvme_fc_lport		*lport;
     	spinlock_t			lock;
     	struct kref			ref;
    +	struct delayed_work		dev_loss_work;
     } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
     
     enum nvme_fcctrl_flags {
    @@ -503,6 +504,8 @@ nvme_fc_free_rport(struct kref *ref)
     	WARN_ON(rport->remoteport.port_state != FC_OBJSTATE_DELETED);
     	WARN_ON(!list_empty(&rport->ctrl_list));
     
    +	cancel_delayed_work_sync(&rport->dev_loss_work);
    +
     	/* remove from lport list */
     	spin_lock_irqsave(&nvme_fc_lock, flags);
     	list_del(&rport->endp_list);
    @@ -530,6 +533,124 @@ nvme_fc_rport_get(struct nvme_fc_rport *rport)
     	return kref_get_unless_zero(&rport->ref);
     }
     
    +static void
    +nvme_fc_resume_controller(struct nvme_fc_ctrl *ctrl)
    +{
    +	switch (ctrl->ctrl.state) {
    +	case NVME_CTRL_NEW:
    +	case NVME_CTRL_RECONNECTING:
    +		/*
    +		 * As all reconnects were suppressed, schedule a
    +		 * connect.
    +		 */
    +		dev_info(ctrl->ctrl.device,
    +			"NVME-FC{%d}: connectivity re-established. "
    +			"Attempting reconnect\n", ctrl->cnum);
    +
    +		queue_delayed_work(nvme_wq, &ctrl->connect_work, 0);
    +		break;
    +
    +	case NVME_CTRL_RESETTING:
    +		/*
    +		 * Controller is already in the process of terminating the
    +		 * association. No need to do anything further. The reconnect
    +		 * step will naturally occur after the reset completes.
    +		 */
    +		break;
    +
    +	default:
    +		/* no action to take - let it delete */
    +		break;
    +	}
    +}
    +
    +static struct nvme_fc_rport *
    +nvme_fc_attach_to_suspended_rport(struct nvme_fc_lport *lport,
    +				struct nvme_fc_port_info *pinfo)
    +{
    +	struct nvme_fc_rport *rport;
    +	struct nvme_fc_ctrl *ctrl;
    +	unsigned long flags;
    +
    +	spin_lock_irqsave(&nvme_fc_lock, flags);
    +
    +	list_for_each_entry(rport, &lport->endp_list, endp_list) {
    +		if (rport->remoteport.node_name != pinfo->node_name ||
    +		    rport->remoteport.port_name != pinfo->port_name)
    +			continue;
    +
    +		if (!nvme_fc_rport_get(rport)) {
    +			rport = ERR_PTR(-ENOLCK);
    +			goto out_done;
    +		}
    +
    +		spin_unlock_irqrestore(&nvme_fc_lock, flags);
    +
    +		cancel_delayed_work_sync(&rport->dev_loss_work);
    +
    +		spin_lock_irqsave(&rport->lock, flags);
    +
    +		/* has it been unregistered */
    +		if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
    +			/* means lldd called us twice */
    +			spin_unlock_irqrestore(&rport->lock, flags);
    +			nvme_fc_rport_put(rport);
    +			return ERR_PTR(-ESTALE);
    +		}
    +
    +		rport->remoteport.port_state = FC_OBJSTATE_ONLINE;
    +
    +		/*
    +		 * kick off a reconnect attempt on all associations to the
    +		 * remote port. A successful reconnects will resume i/o.
    +		 */
    +		list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
    +			nvme_fc_resume_controller(ctrl);
    +
    +		spin_unlock_irqrestore(&rport->lock, flags);
    +
    +		return rport;
    +	}
    +
    +	rport = NULL;
    +
    +out_done:
    +	spin_unlock_irqrestore(&nvme_fc_lock, flags);
    +
    +	return rport;
    +}
    +
    +static void
    +nvme_fc_rport_dev_loss_work(struct work_struct *work)
    +{
    +	struct nvme_fc_rport *rport =
    +			container_of(to_delayed_work(work),
    +				struct nvme_fc_rport, dev_loss_work);
    +	struct nvme_fc_ctrl *ctrl;
    +	unsigned long flags;
    +
    +	spin_lock_irqsave(&rport->lock, flags);
    +
    +	/* If port state transitioned dev loss shouldn't kick in */
    +	if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
    +		spin_unlock_irqrestore(&rport->lock, flags);
    +		return;
    +	}
    +
    +	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
    +		dev_warn(ctrl->ctrl.device,
    +			"NVME-FC{%d}: Remote Port failed to reconnect within "
    +			"dev_loss_tmo (%d seconds). Deleting controller\n",
    +			ctrl->cnum, rport->remoteport.dev_loss_tmo);
    +		if (__nvme_fc_del_ctrl(ctrl))
    +			dev_warn(ctrl->ctrl.device,
    +				"NVME-FC{%d}: delete request failed\n",
    +				ctrl->cnum);
    +	}
    +
    +	spin_unlock_irqrestore(&rport->lock, flags);
    +}
    +
     /**
      * nvme_fc_register_remoteport - transport entry point called by an
      *                              LLDD to register the existence of a NVME
    @@ -556,22 +677,46 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
     	unsigned long flags;
     	int ret, idx;
     
    +	if (!nvme_fc_lport_get(lport)) {
    +		ret = -ESHUTDOWN;
    +		goto out_reghost_failed;
    +	}
    +
    +	/*
    +	 * look to see if there is already a remoteport that is waiting
    +	 * for a reconnect (within dev_loss_tmo) with the same WWN's.
    +	 * If so, transition to it and reconnect.
    +	 */
    +	newrec = nvme_fc_attach_to_suspended_rport(lport, pinfo);
    +
    +	/* found an rport, but something about its state is bad */
    +	if (IS_ERR(newrec)) {
    +		ret = PTR_ERR(newrec);
    +		goto out_lport_put;
    +
    +	/* found existing rport, which was resumed */
    +	} else if (newrec) {
    +		/* Ignore pinfo->dev_loss_tmo. Leave rport and ctlr's as is */
    +
    +		nvme_fc_lport_put(lport);
    +		nvme_fc_signal_discovery_scan(lport, newrec);
    +		*portptr = &newrec->remoteport;
    +		return 0;
    +	}
    +
    +	/* nothing found - allocate a new remoteport struct */
    +
     	newrec = kmalloc((sizeof(*newrec) + lport->ops->remote_priv_sz),
     			 GFP_KERNEL);
     	if (!newrec) {
     		ret = -ENOMEM;
    -		goto out_reghost_failed;
    -	}
    -
    -	if (!nvme_fc_lport_get(lport)) {
    -		ret = -ESHUTDOWN;
    -		goto out_kfree_rport;
    +		goto out_lport_put;
     	}
     
     	idx = ida_simple_get(&lport->endp_cnt, 0, 0, GFP_KERNEL);
     	if (idx < 0) {
     		ret = -ENOSPC;
    -		goto out_lport_put;
    +		goto out_kfree_rport;
     	}
     
     	INIT_LIST_HEAD(&newrec->endp_list);
    @@ -594,6 +739,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
     		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
     	else
     		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
    +	INIT_DELAYED_WORK(&newrec->dev_loss_work, nvme_fc_rport_dev_loss_work);
     
     	spin_lock_irqsave(&nvme_fc_lock, flags);
     	list_add_tail(&newrec->endp_list, &lport->endp_list);
    @@ -604,10 +750,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
     	*portptr = &newrec->remoteport;
     	return 0;
     
    -out_lport_put:
    -	nvme_fc_lport_put(lport);
     out_kfree_rport:
     	kfree(newrec);
    +out_lport_put:
    +	nvme_fc_lport_put(lport);
     out_reghost_failed:
     	*portptr = NULL;
     	return ret;
    @@ -638,6 +784,61 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
     	return 0;
     }
     
    +static void
    +nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
    +{
    +	dev_info(ctrl->ctrl.device,
    +		"NVME-FC{%d}: controller connectivity lost. Awaiting "
    +		"Reconnect", ctrl->cnum);
    +
    +	switch (ctrl->ctrl.state) {
    +	case NVME_CTRL_NEW:
    +	case NVME_CTRL_LIVE:
    +		/*
    +		 * Schedule a controller reset. The reset will
    +		 * terminate the association and schedule the
    +		 * reconnect timer. Reconnects will be attempted
    +		 * until either the ctlr_loss_tmo
    +		 * (max_retries * connect_delay) expires or the
    +		 * remoteport's dev_loss_tmo expires.
    +		 */
    +		if (nvme_reset_ctrl(&ctrl->ctrl)) {
    +			dev_warn(ctrl->ctrl.device,
    +				"NVME-FC{%d}: Couldn't schedule reset. "
    +				"Deleting controller.\n",
    +				ctrl->cnum);
    +			__nvme_fc_del_ctrl(ctrl);
    +		}
    +		break;
    +
    +	case NVME_CTRL_RECONNECTING:
    +		/*
    +		 * The association has already been terminated
    +		 * and the controller is attempting reconnects.
    +		 * No need to do anything futher. Reconnects will
    +		 * be attempted until either the ctlr_loss_tmo
    +		 * (max_retries * connect_delay) expires or the
    +		 * remoteport's dev_loss_tmo expires.
    +		 */
    +		break;
    +
    +	case NVME_CTRL_RESETTING:
    +		/*
    +		 * Controller is already in the process of
    +		 * terminating the association. No need to do
    +		 * anything further. The reconnect step will
    +		 * kick in naturally after the association is
    +		 * terminated.
    +		 */
    +		break;
    +
    +	case NVME_CTRL_DELETING:
    +	default:
    +		/* no action to take - let it delete */
    +		break;
    +	}
    +}
    +
     /**
      * nvme_fc_unregister_remoteport - transport entry point called by an
      *                              LLDD to deregister/remove a previously
    @@ -667,15 +868,32 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
     	}
     	portptr->port_state = FC_OBJSTATE_DELETED;
     
    -	/* tear down all associations to the remote port */
    -	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
    -		__nvme_fc_del_ctrl(ctrl);
    +	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
    +		/* if dev_loss_tmo==0, dev loss is immediate */
    +		if (!portptr->dev_loss_tmo) {
    +			dev_warn(ctrl->ctrl.device,
    +				"NVME-FC{%d}: controller connectivity lost. "
    +				"Deleting controller.\n",
    +				ctrl->cnum);
    +			__nvme_fc_del_ctrl(ctrl);
    +		} else
    +			nvme_fc_ctrl_connectivity_loss(ctrl);
    +	}
     
     	spin_unlock_irqrestore(&rport->lock, flags);
     
     	nvme_fc_abort_lsops(rport);
     
    +	queue_delayed_work(nvme_wq, &rport->dev_loss_work,
    +			portptr->dev_loss_tmo * HZ);
    +
    +	/*
    +	 * release the reference, which will allow, if all controllers
    +	 * go away, which should only occur after dev_loss_tmo occurs,
    +	 * for the rport to be torn down.
    +	 */
     	nvme_fc_rport_put(rport);
    +
     	return 0;
     }
     EXPORT_SYMBOL_GPL(nvme_fc_unregister_remoteport);
    @@ -702,7 +920,6 @@ nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
     			u32 dev_loss_tmo)
     {
     	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
    -	struct nvme_fc_ctrl *ctrl;
     	unsigned long flags;
     
     	spin_lock_irqsave(&rport->lock, flags);
    @@ -2727,25 +2944,31 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
     	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING)
     		return;
     
    -	dev_info(ctrl->ctrl.device,
    -		"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
    -		ctrl->cnum, status);
    +	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
    +		dev_info(ctrl->ctrl.device,
    +			"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
    +			ctrl->cnum, status);
     
     	if (nvmf_should_reconnect(&ctrl->ctrl)) {
    -		/* Only schedule the reconnect if the remote port is online */
    -		if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
    -			return;
    -
    -		dev_info(ctrl->ctrl.device,
    -			"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
    -			ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
    +		if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
    +			dev_info(ctrl->ctrl.device,
    +				"NVME-FC{%d}: Reconnect attempt in %d "
    +				"seconds.\n",
    +				ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
     		queue_delayed_work(nvme_wq, &ctrl->connect_work,
     				ctrl->ctrl.opts->reconnect_delay * HZ);
     	} else {
    -		dev_warn(ctrl->ctrl.device,
    +		if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
    +			dev_warn(ctrl->ctrl.device,
     				"NVME-FC{%d}: Max reconnect attempts (%d) "
     				"reached. Removing controller\n",
     				ctrl->cnum, ctrl->ctrl.nr_reconnects);
    +		else
    +			dev_warn(ctrl->ctrl.device,
    +				"NVME-FC{%d}: Max reconnect attempts (%d) "
    +				"reached while waiting for remoteport "
    +				"connectivity. Removing controller\n",
    +				ctrl->cnum, ctrl->ctrl.nr_reconnects);
     		WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
     	}
     }
    @@ -2769,15 +2992,17 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
     		return;
     	}
     
    -	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) {
    +	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
     		ret = nvme_fc_create_association(ctrl);
    -		if (ret)
    -			nvme_fc_reconnect_or_delete(ctrl, ret);
    -		else
    -			dev_info(ctrl->ctrl.device,
    -				"NVME-FC{%d}: controller reset complete\n",
    -				ctrl->cnum);
    -	}
    +	else
    +		ret = -ENOTCONN;
    +
    +	if (ret)
    +		nvme_fc_reconnect_or_delete(ctrl, ret);
    +	else
    +		dev_info(ctrl->ctrl.device,
    +			"NVME-FC{%d}: controller reset complete\n",
    +			ctrl->cnum);
     }
     
     static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
    -- 
    2.13.1
    
    
    _______________________________________________
    Linux-nvme mailing list
    Linux-nvme at lists.infradead.org
    http://lists.infradead.org/mailman/listinfo/linux-nvme
    


From jsmart2021 at gmail.com  Thu Oct 19 08:58:13 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 08:58:13 -0700
Subject: [PATCH 1/3] nvme: add duplicate_connect option
In-Reply-To: <20171019152024.GA4239@infradead.org>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
 <20171011003512.5946-2-jsmart2021@gmail.com>
 <20171019152024.GA4239@infradead.org>
Message-ID: <458ccb0c-41fd-2709-5d90-033f8a111e75@gmail.com>

On 10/19/2017 8:20 AM, Christoph Hellwig wrote:
>> +	{ NVMF_OPT_DUP_CONNECT,		"duplicate_connect=%d"	},
> 
> This is a boolean parameter, so please drop the "=%d here".
> 
>> +		case NVMF_OPT_DUP_CONNECT:
>> +			if (match_int(args, &token)) {
>> +				ret = -EINVAL;
>> +				goto out;
>> +			}
>> +			if (token == 0)
>> +				opts->duplicate_connect = false;
>> +			else if (token == 1)
>> +				opts->duplicate_connect = true;
>> +			else {
>> +				pr_err("Invalid duplicate_connect %d\n", token);
>> +				ret = -EINVAL;
>> +				goto out;
>> +			}
>> +			break;
> 
> And replace all this code by:
> 
> 	case NVMF_OPT_DUP_CONNECT:
> 		opts->duplicate_connect = true;
> 		break;
> 
> That being said I'd be tempted to keep the existing behavior as
> the default, and make noduplicates the option.
> 

ok, so the mere existence of the parameter means it's true.  That's fine 
and I can repost.

I disagree with allowing duplicates to be the default.  Generically, I 
think it should be the exception. Let the admin that will create 
duplicates, thus is well aware of the ramifications and that it is a 
duplication, use the explicit syntax.  Note: currently patches only 
allow the option for "nvme connect", not "connect-all".

Selfishly, making duplicates allowed the defaults also impacts the 
scripts posted for FC, and widens the nvme cli syntax to connect-all.

-- james


From jsmart2021 at gmail.com  Thu Oct 19 08:59:06 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 08:59:06 -0700
Subject: [PATCH 2/3] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171019152317.GB4239@infradead.org>
References: <20171011003512.5946-1-jsmart2021@gmail.com>
 <20171011003512.5946-3-jsmart2021@gmail.com>
 <20171019152317.GB4239@infradead.org>
Message-ID: <89d38cc7-0a6e-d374-4005-96a178a18c78@gmail.com>

On 10/19/2017 8:23 AM, Christoph Hellwig wrote:
>> +static inline bool
>> +__nvme_fc_options_match(struct nvmf_ctrl_options *opts,
>> +			struct nvme_fc_ctrl *ctrl)
>> +{
>> +	if (strcmp(opts->subsysnqn, ctrl->ctrl.opts->subsysnqn) ||
>> +	    strcmp(opts->host->nqn, ctrl->ctrl.opts->host->nqn) ||
>> +	    memcmp(&opts->host->id, &ctrl->ctrl.opts->host->id,
>> +				sizeof(uuid_t)))
>> +		return false;
> 
> This just checks generic options and isn't FC specific.  As a properly
> named helper it might make sense in common code.
> 

Agree, I can make a short helper

-- james



From jsmart2021 at gmail.com  Thu Oct 19 11:04:38 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 11:04:38 -0700
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <32B40236-CB8E-461E-B765-AEF544AB981A@cavium.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-6-jsmart2021@gmail.com>
 <32B40236-CB8E-461E-B765-AEF544AB981A@cavium.com>
Message-ID: <5c86018d-79f5-20d3-07d5-64bdd0344e34@gmail.com>

On 10/19/2017 8:46 AM, Trapp, Darren wrote:
> It seems like we need a call back to the LLD to let him know that the dev_loss_tmo has been canceled upon device coming back.

I don't think so. We already have the desired functionality in the 
remoteport_delete callback.

In SCSI we had devloss tied 1:1 with the remote_port, devloss started 
when remote_port unregistered, and when dev_loss_tmo fired thus 
releasing the remote port, we called the dev_loss_tmo_callbk. So the 
dev_loss_tmo_callbk is actually the remoteport delete callback. There 
was also the notion that the SCSI target, which is 1:1 with the remote 
port, was visible on the os until that callback occurred.

In nvme-fc, the lldd calls the unregister remoteport, and when the 
remoteport is finished/done, the remoteport_delete will be called back.
Currently that callback is based on the remoteport datastructure being 
freed which means it waits for all controllers to be freed as well.
This turns out to be rather ugly as controller freeing is based on 
namespace devices being freed, and some apps can hold on to the devnode 
a long time. So I've been wrapping up a patch that instead has the 
remoteport_delete called when all associations relative to the lldd have 
been terminated.  The association terminations are done immediately upon 
the unregister call. That should have killed all hw queues, etc leaving 
nothing left for the lldd to be tracking. Although the transport<->lldd 
associations are killed, the nvme controller and it's namespaces are 
still live relative to the os, just stalled waiting for the 
ctrl_loss_tmo or dev_loss_tmo to fire. Those timers can be/are 
completely asynchronous to the remoteport_delete.  If the lldd registers 
a new and matching remoteport, the transport will reuse the existing 
remoteport structures and related controllers, etc - just updating the 
LLDD related info (which I've found a bug or two in).

Is there a specific reason you would want to hold off the remoteport 
delete for a full dev_loss_tmo vs as soon as the transport terminates 
the references to the lldd relative to remoteport (e.g. association 
terminations)?   With the association teardowns only, it can be much 
faster than dev_loss_tmo.


> 
> The LLD has called the unregister_remoteport in the transport and done a wait_for_completion.  If the device re-appears before dev_loss_tmo has been reached, the transport cancels the del_work.  But, the LLD still has the completion event pending.

Ahh. I understand. The wait_for_completions are in the lldd and I know 
older code shows had the sequence you describe. True, if we reuse the 
remoteport struct, as references never expired, the delete call would 
have never occurred. So this is a bug with what's there.

I think the cleanest fix is to have the delete callback occur as soon as 
the delete_associations are done - as that will always occur and do so 
rather quickly. If needed, if we find we are reusing a remoteport struct 
and we hadn't called the delete, we call it prior to returning to the 
second register remoteport call, which should work as well.

note: as when we call remoteport_delete (ref cnt and free vs association 
terminate) is independent from dev_loss_tmo, it shouldn't affect this 
dev_loss patch set.

-- james


From Darren.Trapp at cavium.com  Thu Oct 19 12:49:08 2017
From: Darren.Trapp at cavium.com (Trapp, Darren)
Date: Thu, 19 Oct 2017 19:49:08 +0000
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <5c86018d-79f5-20d3-07d5-64bdd0344e34@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-6-jsmart2021@gmail.com>
 <32B40236-CB8E-461E-B765-AEF544AB981A@cavium.com>
 <5c86018d-79f5-20d3-07d5-64bdd0344e34@gmail.com>
Message-ID: <925E24E5-10DF-4E72-ADB7-F12A64E93655@cavium.com>

Thanks for the response. I think if it tears down the rport immediately and always calls back to the LLD it solves the problem.



On 10/19/17, 11:04 AM, "James Smart" <jsmart2021 at gmail.com> wrote:


    Ahh. I understand. The wait_for_completions are in the lldd and I know 
    older code shows had the sequence you describe. True, if we reuse the 
    remoteport struct, as references never expired, the delete call would 
    have never occurred. So this is a bug with what's there.
    
    I think the cleanest fix is to have the delete callback occur as soon as 
    the delete_associations are done - as that will always occur and do so 
    rather quickly. If needed, if we find we are reusing a remoteport struct 
    and we hadn't called the delete, we call it prior to returning to the 
    second register remoteport call, which should work as well.
    
    note: as when we call remoteport_delete (ref cnt and free vs association 
    terminate) is independent from dev_loss_tmo, it shouldn't affect this 
    dev_loss patch set.
    
    -- james
    


From jsmart2021 at gmail.com  Thu Oct 19 13:02:36 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 13:02:36 -0700
Subject: [REPOST PATCH] nvmet: Fix fatal_err_work deadlock
Message-ID: <20171019200236.15892-1-jsmart2021@gmail.com>

Below is  a stack trace for an issue that was reported.

What's happening is that the nvmet layer had it's controller kato
timeout fire, which causes it to schedule its fatal error handler
via the fatal_err_work element. The error handler is invoked, which
calls the transport delete_ctrl() entry point, and as the transport
tears down the controller, nvmet_sq_destroy ends up doing the final
put on the ctlr causing it to enter its free routine. The ctlr free
routine does a cancel_work_sync() on fatal_err_work element, which
then does a flush_work and wait_for_completion. But, as the wait is
in the context of the work element being flushed, its in a catch-22
and the thread hangs.

[  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  326.909832] nvmet: ctrl 1 fatal error occurred!
[  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
x114 flg x2
[  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
seconds.
[  494.589638]       Not tainted 4.14.0-rc1.James+ #1
[  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  494.603718] kworker/0:2     D    0   243      2 0x80000000
[  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
[  494.616447] Call Trace:
[  494.619177]  __schedule+0x28d/0x890
[  494.623070]  schedule+0x36/0x80
[  494.626571]  schedule_timeout+0x1dd/0x300
[  494.631044]  ? dequeue_task_fair+0x592/0x840
[  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
[  494.640756]  wait_for_completion+0x121/0x180
[  494.645521]  ? wake_up_q+0x80/0x80
[  494.649315]  flush_work+0x11d/0x1a0
[  494.653206]  ? wake_up_worker+0x30/0x30
[  494.657484]  __cancel_work_timer+0x10b/0x190
[  494.662249]  cancel_work_sync+0x10/0x20
[  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
[  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
[  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
[  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
[  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
[  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
[  494.701589]  process_one_work+0x149/0x360
[  494.706064]  worker_thread+0x4d/0x3c0
[  494.710148]  kthread+0x109/0x140
[  494.713751]  ? rescuer_thread+0x380/0x380
[  494.718214]  ? kthread_park+0x60/0x60

Correct by creating a final free work element, which is scheduled
by the final ctrl put routine, so that when the cancel_work_sync
is called, it cannot be in the same work_q element context.

Signed-off-by: James Smart <james.smart at broadcom.com>
Acked-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/target/core.c  | 26 ++++++++++++++++++--------
 drivers/nvme/target/nvmet.h |  1 +
 2 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 1b208beeef50..cfc35cbb6fe2 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -840,17 +840,12 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	return status;
 }
 
-static void nvmet_ctrl_free(struct kref *ref)
+static void nvmet_ctrl_free_work(struct work_struct *work)
 {
-	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_ctrl *ctrl =
+			container_of(work, struct nvmet_ctrl, free_work);
 	struct nvmet_subsys *subsys = ctrl->subsys;
 
-	nvmet_stop_keep_alive_timer(ctrl);
-
-	mutex_lock(&subsys->lock);
-	list_del(&ctrl->subsys_entry);
-	mutex_unlock(&subsys->lock);
-
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fatal_err_work);
 
@@ -862,6 +857,21 @@ static void nvmet_ctrl_free(struct kref *ref)
 	kfree(ctrl);
 }
 
+static void nvmet_ctrl_free(struct kref *ref)
+{
+	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_subsys *subsys = ctrl->subsys;
+
+	nvmet_stop_keep_alive_timer(ctrl);
+
+	mutex_lock(&subsys->lock);
+	list_del(&ctrl->subsys_entry);
+	mutex_unlock(&subsys->lock);
+
+	INIT_WORK(&ctrl->free_work, nvmet_ctrl_free_work);
+	schedule_work(&ctrl->free_work);
+}
+
 void nvmet_ctrl_put(struct nvmet_ctrl *ctrl)
 {
 	kref_put(&ctrl->ref, nvmet_ctrl_free);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index e342f02845c1..ed38b44a7007 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -129,6 +129,7 @@ struct nvmet_ctrl {
 	struct kref		ref;
 	struct delayed_work	ka_work;
 	struct work_struct	fatal_err_work;
+	struct work_struct	free_work;
 
 	struct nvmet_fabrics_ops *ops;
 
-- 
2.13.1



From james.smart at broadcom.com  Thu Oct 19 13:24:37 2017
From: james.smart at broadcom.com (James Smart)
Date: Thu, 19 Oct 2017 13:24:37 -0700
Subject: [PATCH] nvme_fc: avoid workqueue flush stalls
In-Reply-To: <01c52881-b172-0d86-5db2-596acf6394db@grimberg.me>
References: <20170928165631.16097-1-jsmart2021@gmail.com>
 <01c52881-b172-0d86-5db2-596acf6394db@grimberg.me>
Message-ID: <327c6f13-0bd3-5e95-a9e5-d104c561680d@broadcom.com>

On 9/30/2017 10:18 AM, Sagi Grimberg wrote:
>
>> There's no need to wait for the full nvme_wq, which is now shared,
>> to flush. flush only the delete_work item.
>
> The patch looks correct, but can you explain what stalled that this
> patch fixes?

I don't have an explicit failure case. While reviewing the code for the 
other stall issue with fatal_err_work, I realized that this flush is now 
waiting on a work queue shared by everything nvmf.? It used to be solely 
a nvme-fc transport workqueue. There's no need for waiting on anything 
more than the delete work item. And having it raise in scope of the nvmf 
thread perhaps makes it wait for a lot of other unnecessary things - 
slowing it down.

If you're ok with it - can you Ack/Review it ?

-- james




From jsmart2021 at gmail.com  Thu Oct 19 16:11:37 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 16:11:37 -0700
Subject: [PATCH v2 0/2] nvme_fc: rework io timeouts and termination
Message-ID: <20171019231139.23611-1-jsmart2021@gmail.com>

This is a rework of a prior patch that has now been split into two
patches. One patch addresses changes to the io timeout error handler.
The other patch addresses io completions for failing ios to ensure
io retries or to fail commands during association create.

Patches also address prior review comments

James Smart (2):
  nvme_fc: correct io termination handling
  nvme_fc: correct io timeout behavior

 drivers/nvme/host/fc.c | 45 ++++++++++++++++++++++++++++-----------------
 1 file changed, 28 insertions(+), 17 deletions(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Thu Oct 19 16:11:38 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 16:11:38 -0700
Subject: [PATCH v2 1/2] nvme_fc: correct io termination handling
In-Reply-To: <20171019231139.23611-1-jsmart2021@gmail.com>
References: <20171019231139.23611-1-jsmart2021@gmail.com>
Message-ID: <20171019231139.23611-2-jsmart2021@gmail.com>

The io completion handling for i/o's that are failing due to
to a transport error or association termination had issues, causing
io failures (DNR set so retries didn't kick in) or long stalls.

Change the io completion handler for the following items:

When an io has been completed due to a transport abort (based on an
exchange error) or when marked as aborted as part of an association
termination (FCOP_FLAGS_TERMIO), set the NVME completion status to
NVME_SC_ABORTED. By default, do not set DNR on the status so that a
retry can be attempted after association recreate.

In cases where an io is failed (non-successful nvme status including
aborted), if the controller is being deleted (blk_queue_dying) or
the io was part of the ios used for association creation (ctrl state
is NEW or RECONNECTING), then additionally set the DNR bit so the io
will not be retried. If the failed io was part of association creation,
the failure will tear down the partially completioned association and
typically restart a new reconnect attempt (another create association
later).

Rearranged code flow to remove a largely unneeded local variable.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 18e036c6833b..ce8617188d3a 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -1387,7 +1387,7 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
 	struct nvme_command *sqe = &op->cmd_iu.sqe;
 	__le16 status = cpu_to_le16(NVME_SC_SUCCESS << 1);
 	union nvme_result result;
-	bool complete_rq, terminate_assoc = true;
+	bool terminate_assoc = true;
 
 	/*
 	 * WARNING:
@@ -1429,8 +1429,9 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
 	fc_dma_sync_single_for_cpu(ctrl->lport->dev, op->fcp_req.rspdma,
 				sizeof(op->rsp_iu), DMA_FROM_DEVICE);
 
-	if (atomic_read(&op->state) == FCPOP_STATE_ABORTED)
-		status = cpu_to_le16((NVME_SC_ABORT_REQ | NVME_SC_DNR) << 1);
+	if (atomic_read(&op->state) == FCPOP_STATE_ABORTED ||
+			op->flags & FCOP_FLAGS_TERMIO)
+		status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);
 	else if (freq->status)
 		status = cpu_to_le16(NVME_SC_INTERNAL << 1);
 
@@ -1494,23 +1495,27 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
 done:
 	if (op->flags & FCOP_FLAGS_AEN) {
 		nvme_complete_async_event(&queue->ctrl->ctrl, status, &result);
-		complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
+		__nvme_fc_fcpop_chk_teardowns(ctrl, op);
 		atomic_set(&op->state, FCPOP_STATE_IDLE);
 		op->flags = FCOP_FLAGS_AEN;	/* clear other flags */
 		nvme_fc_ctrl_put(ctrl);
 		goto check_error;
 	}
 
-	complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
-	if (!complete_rq) {
-		if (unlikely(op->flags & FCOP_FLAGS_TERMIO)) {
-			status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);
-			if (blk_queue_dying(rq->q))
-				status |= cpu_to_le16(NVME_SC_DNR << 1);
-		}
-		nvme_end_request(rq, status, result);
-	} else
+	/*
+	 * Force failures of commands if we're killing the controller
+	 * or have an error on a command used to create an new association
+	 */
+	if (status &&
+	    (blk_queue_dying(rq->q) ||
+	     ctrl->ctrl.state == NVME_CTRL_NEW ||
+	     ctrl->ctrl.state == NVME_CTRL_RECONNECTING))
+		status |= cpu_to_le16(NVME_SC_DNR << 1);
+
+	if (__nvme_fc_fcpop_chk_teardowns(ctrl, op))
 		__nvme_fc_final_op_cleanup(rq);
+	else
+		nvme_end_request(rq, status, result);
 
 check_error:
 	if (terminate_assoc)
-- 
2.13.1



From jsmart2021 at gmail.com  Thu Oct 19 16:11:39 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 19 Oct 2017 16:11:39 -0700
Subject: [PATCH v2 2/2] nvme_fc: correct io timeout behavior
In-Reply-To: <20171019231139.23611-1-jsmart2021@gmail.com>
References: <20171019231139.23611-1-jsmart2021@gmail.com>
Message-ID: <20171019231139.23611-3-jsmart2021@gmail.com>

The transport io timeout behavior wasn't quite correct. It ignored
that the io error handler is supposed to be synchronous so it possibly
allowed the blk request to be restarted while the io associated was
still aborting. Timeouts on reserved commands, those used for
association create, were never timing out thus they hung out forever.

To correct:
If an io is times out while a remoteport is not connected, just
restart the io timer. The lack of connectivity will simultaneously
be resetting the controller, so the reset path will abort and terminate
the io.

If an io is times out while it was marked for transport abort, just
reset the io timer. The abort process is underway and will complete
the io.

Otherwise, if an io times out, abort the io. If the abort was
unsuccessful (unlikely) give up and return not handled.

If the abort was successful, as the abort process is underway it will
terminate the io, so rather than synchronously waiting, just restart
the io timer.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index ce8617188d3a..78091bc7af69 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -1903,13 +1903,14 @@ nvme_fc_timeout(struct request *rq, bool reserved)
 	struct nvme_fc_ctrl *ctrl = op->ctrl;
 	int ret;
 
-	if (reserved)
+	if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE ||
+			atomic_read(&op->state) == FCPOP_STATE_ABORTED)
 		return BLK_EH_RESET_TIMER;
 
 	ret = __nvme_fc_abort_op(ctrl, op);
 	if (ret)
-		/* io wasn't active to abort consider it done */
-		return BLK_EH_HANDLED;
+		/* io wasn't active to abort */
+		return BLK_EH_NOT_HANDLED;
 
 	/*
 	 * we can't individually ABTS an io without affecting the queue,
@@ -1920,7 +1921,12 @@ nvme_fc_timeout(struct request *rq, bool reserved)
 	 */
 	nvme_fc_error_recovery(ctrl, "io timeout error");
 
-	return BLK_EH_HANDLED;
+	/*
+	 * the io abort has been initiated. Have the reset timer
+	 * restarted and the abort completion will complete the io
+	 * shortly. Avoids a synchronous wait while the abort finishes.
+	 */
+	return BLK_EH_RESET_TIMER;
 }
 
 static int
-- 
2.13.1



From hare at suse.de  Thu Oct 19 22:58:58 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 07:58:58 +0200
Subject: [PATCH v3 1/5] nvme core: allow controller RESETTING to
 RECONNECTING transition
In-Reply-To: <20171017233248.6769-2-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-2-jsmart2021@gmail.com>
Message-ID: <72657ea3-d42e-0fdc-a5c6-9bdd9f47237b@suse.de>

On 10/18/2017 01:32 AM, James Smart wrote:
> Allow controller state transition : RESETTING to RECONNECTING
> 
> Transport will typically transition from LIVE->RESETTING when
> initially performing a reset or recovering from an error. Adding
> this transition allows a transport to transition to RECONNECTING
> when it checks/waits for connectivity then creates new transport
> connections and reinits the controller.
> 
> -- james
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> Reviewed-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> 
> ---
> v3: no change. incorporated Reviewed-by's
> ---
>  drivers/nvme/host/core.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 26c8913435b2..aa9e2df27bf7 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -205,6 +205,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
>  	case NVME_CTRL_RECONNECTING:
>  		switch (old_state) {
>  		case NVME_CTRL_LIVE:
> +		case NVME_CTRL_RESETTING:
>  			changed = true;
>  			/* FALLTHRU */
>  		default:
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Thu Oct 19 23:00:08 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 08:00:08 +0200
Subject: [PATCH v3 2/5] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20171017233248.6769-3-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-3-jsmart2021@gmail.com>
Message-ID: <06d12d08-990f-26fa-fe73-127dbe4a7f4e@suse.de>

On 10/18/2017 01:32 AM, James Smart wrote:
> Clean up some of the controller state checks and add the
> RESETTING->RECONNECTING state transition.
> 
> Specifically:
> - the movement of the RESETTING state change and schedule of
>   reset_work to core doesn't work wiht nvme_fc_error_recovery
>   setting state to RECONNECTING before attempting to reset.
>   Remove the state change as the reset request does it.
> - In the rare cases where an error occurs right as we're
>   transitioning to LIVE, defer the controller start actions.
> - In error handling on teardown of associations while
>   performing initial controller creation - avoid quiesce
>   calls on the admin_q. They are unneeded.
> - Add the RESETTING->RECONNECTING transition in the reset
>   handler.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> Reviewed-by: Christoph Hellwig <hch at lst.de>
> 
> ---
> v3: no change. incorporated Reviewed-by's
> ---
>  drivers/nvme/host/fc.c | 28 +++++++++++++---------------
>  1 file changed, 13 insertions(+), 15 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Thu Oct 19 23:04:02 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 08:04:02 +0200
Subject: [PATCH v3 3/5] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <20171017233248.6769-4-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-4-jsmart2021@gmail.com>
Message-ID: <ee0d6875-843f-103e-fa1e-825bf18acf48@suse.de>

On 10/18/2017 01:32 AM, James Smart wrote:
> Add a dev_loss_tmo value, paralleling the SCSI FC transport, for device
> connectivity loss.
> 
> The transport initializes the value in the nvme_fc_register_remoteport()
> call. If the value is not set, a default of 60s is set.
> 
> Add a new routine to the api, nvme_fc_set_remoteport_devloss() routine,
> which allows the lldd to dynamically update the value on an existing
> remoteport.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> 
> ---
> v3:
>  Removed the expected reconnect time and min devloss defines. They
>  were not that meaningful.
>  Incorporated the new nvme_fc_set_remoteport_devloss() routine
>  which was in a different patch in v2. New routine no longer
>  touches controllers. It only modifies the remoteport.
> ---
>  drivers/nvme/host/fc.c         | 31 +++++++++++++++++++++++++++++++
>  include/linux/nvme-fc-driver.h | 11 +++++++++--
>  2 files changed, 40 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 5afb518c39ad..3ac49e670c38 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -45,6 +45,8 @@ enum nvme_fc_queue_flags {
>  
>  #define NVMEFC_QUEUE_DELAY	3		/* ms units */
>  
> +#define NVME_FC_DEFAULT_DEV_LOSS_TMO	60	/* seconds */
> +
>  struct nvme_fc_queue {
>  	struct nvme_fc_ctrl	*ctrl;
>  	struct device		*dev;
> @@ -587,6 +589,11 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
>  	newrec->remoteport.port_id = pinfo->port_id;
>  	newrec->remoteport.port_state = FC_OBJSTATE_ONLINE;
>  	newrec->remoteport.port_num = idx;
> +	/* a registration value of dev_loss_tmo=0 results in the default */
> +	if (pinfo->dev_loss_tmo)
> +		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
> +	else
> +		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
>  
>  	spin_lock_irqsave(&nvme_fc_lock, flags);
>  	list_add_tail(&newrec->endp_list, &lport->endp_list);
> @@ -690,6 +697,30 @@ nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport)
>  }
>  EXPORT_SYMBOL_GPL(nvme_fc_rescan_remoteport);
>  
> +int
> +nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
> +			u32 dev_loss_tmo)
> +{
> +	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
> +	struct nvme_fc_ctrl *ctrl;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&rport->lock, flags);
> +
> +	if (portptr->port_state != FC_OBJSTATE_ONLINE) {
> +		spin_unlock_irqrestore(&rport->lock, flags);
> +		return -EINVAL;
> +	}
> +
> +	/* a dev_loss_tmo of 0 (immediate) is allowed to be set */
> +	rport->remoteport.dev_loss_tmo = dev_loss_tmo;
> +
> +	spin_unlock_irqrestore(&rport->lock, flags);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(nvme_fc_set_remoteport_devloss);
> +
>  
>  /* *********************** FC-NVME DMA Handling **************************** */
>  
> diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
> index 4ea03b9a5c8c..3de268bf86bf 100644
> --- a/include/linux/nvme-fc-driver.h
> +++ b/include/linux/nvme-fc-driver.h
> @@ -40,6 +40,8 @@
>   * @node_name: FC WWNN for the port
>   * @port_name: FC WWPN for the port
>   * @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx)
> + * @dev_loss_tmo: maximum delay for reconnects to an association on
> + *             this device. Used only on a remoteport.
>   *
>   * Initialization values for dynamic port fields:
>   * @port_id:      FC N_Port_ID currently assigned the port. Upper 8 bits must
> @@ -50,6 +52,7 @@ struct nvme_fc_port_info {
>  	u64			port_name;
>  	u32			port_role;
>  	u32			port_id;
> +	u32			dev_loss_tmo;
>  };
>  
>  
> @@ -202,6 +205,9 @@ enum nvme_fc_obj_state {
>   *             The length of the buffer corresponds to the local_priv_sz
>   *             value specified in the nvme_fc_port_template supplied by
>   *             the LLDD.
> + * @dev_loss_tmo: maximum delay for reconnects to an association on
> + *             this device. To modify, lldd must call
> + *             nvme_fc_set_remoteport_devloss().
>   *
>   * Fields with dynamic values. Values may change base on link state. LLDD
>   * may reference fields directly to change them. Initialized by the
> @@ -259,10 +265,9 @@ struct nvme_fc_remote_port {
>  	u32 port_role;
>  	u64 node_name;
>  	u64 port_name;
> -
>  	struct nvme_fc_local_port *localport;
> -
>  	void *private;
> +	u32 dev_loss_tmo;
>  
>  	/* dynamic fields */
>  	u32 port_id;
> @@ -448,6 +453,8 @@ int nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *remoteport);
>  
>  void nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport);
>  
> +int nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *remoteport,
> +			u32 dev_loss_tmo);
>  
>  
>  /*
> 
Is this exported to sysfs somewhere? If not, can we have it in sysfs?
It surely made live _so_ much easier when configuring the odd device...

Other than that:

Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Thu Oct 19 23:05:00 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 08:05:00 +0200
Subject: [PATCH v3 4/5] nvme_fc: check connectivity before initiating
 reconnects
In-Reply-To: <20171017233248.6769-5-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-5-jsmart2021@gmail.com>
Message-ID: <2f0c685c-1a9d-7adf-0177-0cdee58a3a8b@suse.de>

On 10/18/2017 01:32 AM, James Smart wrote:
> check remoteport connectivity before initiating reconnects
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> 
> ---
> v3:
>   removed inline with lock around 1 line read of state.
>   replaced calls with explicit checks of read state
> ---
>  drivers/nvme/host/fc.c | 23 ++++++++++++++++-------
>  1 file changed, 16 insertions(+), 7 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Thu Oct 19 23:27:55 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 08:27:55 +0200
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20171017233248.6769-6-jsmart2021@gmail.com>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-6-jsmart2021@gmail.com>
Message-ID: <1ea7f564-36ed-a952-22c3-4f7b58be6665@suse.de>

On 10/18/2017 01:32 AM, James Smart wrote:
> This patch adds the dev_loss_tmo functionality to the transport.
> 
> When a remoteport is unregistered (connectivity lost), the following
> actions are taken:
> - the remoteport is marked DELETED
> - a dev_loss_tmo timer is started for the remoteport
> - all controllers on the remoteport are reset.
> 
> After a controller resets, it will stall in a RECONNECTING state
> waiting for one of the following:
> - the controller will continue to attempt reconnect per max_retries
>   and reconnect_delay. As no remoteport connectivity, the reconnect
>   attempt will immediately fail. if max reconnect attempts are
>   reached (e.g. ctrl_loss_tmo reached) the controller is deleted.
> - the remoteport is re-registered prior to dev_loss_tmo expiring.
>   The resume of the remoteport will immediately attempt to reconnect
>   each of its suspended controllers.
> - the remoteport's dev_loss_tmo expires, causing all of its
>   controllers to be deleted.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> 
> v3:
>   Reworked so dev_loss_tmo specific to the remoteport. Revised
>   so connectivity loss resets controllers, and connectivity gain
>   schedules reconnect.
> ---
>  drivers/nvme/host/fc.c | 289 +++++++++++++++++++++++++++++++++++++++++++------
>  1 file changed, 257 insertions(+), 32 deletions(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 61da2f92f71a..3628790371d7 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -138,6 +138,7 @@ struct nvme_fc_rport {
>  	struct nvme_fc_lport		*lport;
>  	spinlock_t			lock;
>  	struct kref			ref;
> +	struct delayed_work		dev_loss_work;
>  } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
>  
>  enum nvme_fcctrl_flags {
> @@ -503,6 +504,8 @@ nvme_fc_free_rport(struct kref *ref)
>  	WARN_ON(rport->remoteport.port_state != FC_OBJSTATE_DELETED);
>  	WARN_ON(!list_empty(&rport->ctrl_list));
>  
> +	cancel_delayed_work_sync(&rport->dev_loss_work);
> +
>  	/* remove from lport list */
>  	spin_lock_irqsave(&nvme_fc_lock, flags);
>  	list_del(&rport->endp_list);
> @@ -530,6 +533,124 @@ nvme_fc_rport_get(struct nvme_fc_rport *rport)
>  	return kref_get_unless_zero(&rport->ref);
>  }
>  
> +static void
> +nvme_fc_resume_controller(struct nvme_fc_ctrl *ctrl)
> +{
> +	switch (ctrl->ctrl.state) {
> +	case NVME_CTRL_NEW:
> +	case NVME_CTRL_RECONNECTING:
> +		/*
> +		 * As all reconnects were suppressed, schedule a
> +		 * connect.
> +		 */
> +		dev_info(ctrl->ctrl.device,
> +			"NVME-FC{%d}: connectivity re-established. "
> +			"Attempting reconnect\n", ctrl->cnum);
> +
> +		queue_delayed_work(nvme_wq, &ctrl->connect_work, 0);
> +		break;
> +
> +	case NVME_CTRL_RESETTING:
> +		/*
> +		 * Controller is already in the process of terminating the
> +		 * association. No need to do anything further. The reconnect
> +		 * step will naturally occur after the reset completes.
> +		 */
> +		break;
> +
> +	default:
> +		/* no action to take - let it delete */
> +		break;
> +	}
> +}
> +
> +static struct nvme_fc_rport *
> +nvme_fc_attach_to_suspended_rport(struct nvme_fc_lport *lport,
> +				struct nvme_fc_port_info *pinfo)
> +{
> +	struct nvme_fc_rport *rport;
> +	struct nvme_fc_ctrl *ctrl;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&nvme_fc_lock, flags);
> +
> +	list_for_each_entry(rport, &lport->endp_list, endp_list) {
> +		if (rport->remoteport.node_name != pinfo->node_name ||
> +		    rport->remoteport.port_name != pinfo->port_name)
> +			continue;
> +
> +		if (!nvme_fc_rport_get(rport)) {
> +			rport = ERR_PTR(-ENOLCK);
> +			goto out_done;
> +		}
> +
> +		spin_unlock_irqrestore(&nvme_fc_lock, flags);
> +
> +		cancel_delayed_work_sync(&rport->dev_loss_work);
> +
> +		spin_lock_irqsave(&rport->lock, flags);
> +
> +		/* has it been unregistered */
> +		if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
> +			/* means lldd called us twice */
> +			spin_unlock_irqrestore(&rport->lock, flags);
> +			nvme_fc_rport_put(rport);
> +			return ERR_PTR(-ESTALE);
> +		}
> +
Why can't we re-use this rport?
It's patently _not_ deleted, so we should be good to use it, right?
Someone else might've messed up, but if _we_ get our reference counting
right that shouldn't affect us, no?
Plus: How would we recover from such a situation?
The rport most likely will _never_ go away, meaning we'll always hit
this scenario for each and every connection attempt.
(Speaking from experience here; I've hit this scenario accidentally :-)
So the only chance we have is a reboot.

> +		rport->remoteport.port_state = FC_OBJSTATE_ONLINE;
> +
> +		/*
> +		 * kick off a reconnect attempt on all associations to the
> +		 * remote port. A successful reconnects will resume i/o.
> +		 */
> +		list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
> +			nvme_fc_resume_controller(ctrl);
> +
> +		spin_unlock_irqrestore(&rport->lock, flags);
> +
> +		return rport;
> +	}
> +
> +	rport = NULL;
> +
> +out_done:
> +	spin_unlock_irqrestore(&nvme_fc_lock, flags);
> +
> +	return rport;
> +}
> +
> +static void
> +nvme_fc_rport_dev_loss_work(struct work_struct *work)
> +{
> +	struct nvme_fc_rport *rport =
> +			container_of(to_delayed_work(work),
> +				struct nvme_fc_rport, dev_loss_work);
> +	struct nvme_fc_ctrl *ctrl;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&rport->lock, flags);
> +
> +	/* If port state transitioned dev loss shouldn't kick in */
> +	if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
> +		spin_unlock_irqrestore(&rport->lock, flags);
> +		return;
> +	}
> +
> +	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
> +		dev_warn(ctrl->ctrl.device,
> +			"NVME-FC{%d}: Remote Port failed to reconnect within "
> +			"dev_loss_tmo (%d seconds). Deleting controller\n",
> +			ctrl->cnum, rport->remoteport.dev_loss_tmo);
> +		if (__nvme_fc_del_ctrl(ctrl))
> +			dev_warn(ctrl->ctrl.device,
> +				"NVME-FC{%d}: delete request failed\n",
> +				ctrl->cnum);
> +	}
> +
> +	spin_unlock_irqrestore(&rport->lock, flags);
> +}
> +
>  /**
>   * nvme_fc_register_remoteport - transport entry point called by an
>   *                              LLDD to register the existence of a NVME
> @@ -556,22 +677,46 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
>  	unsigned long flags;
>  	int ret, idx;
>  
> +	if (!nvme_fc_lport_get(lport)) {
> +		ret = -ESHUTDOWN;
> +		goto out_reghost_failed;
> +	}
> +
> +	/*
> +	 * look to see if there is already a remoteport that is waiting
> +	 * for a reconnect (within dev_loss_tmo) with the same WWN's.
> +	 * If so, transition to it and reconnect.
> +	 */
> +	newrec = nvme_fc_attach_to_suspended_rport(lport, pinfo);
> +
> +	/* found an rport, but something about its state is bad */
> +	if (IS_ERR(newrec)) {
> +		ret = PTR_ERR(newrec);
> +		goto out_lport_put;
> +
> +	/* found existing rport, which was resumed */
> +	} else if (newrec) {
> +		/* Ignore pinfo->dev_loss_tmo. Leave rport and ctlr's as is */
> +
> +		nvme_fc_lport_put(lport);
> +		nvme_fc_signal_discovery_scan(lport, newrec);
> +		*portptr = &newrec->remoteport;
> +		return 0;
> +	}
> +
> +	/* nothing found - allocate a new remoteport struct */
> +
>  	newrec = kmalloc((sizeof(*newrec) + lport->ops->remote_priv_sz),
>  			 GFP_KERNEL);
>  	if (!newrec) {
>  		ret = -ENOMEM;
> -		goto out_reghost_failed;
> -	}
> -
> -	if (!nvme_fc_lport_get(lport)) {
> -		ret = -ESHUTDOWN;
> -		goto out_kfree_rport;
> +		goto out_lport_put;
>  	}
>  
>  	idx = ida_simple_get(&lport->endp_cnt, 0, 0, GFP_KERNEL);
>  	if (idx < 0) {
>  		ret = -ENOSPC;
> -		goto out_lport_put;
> +		goto out_kfree_rport;
>  	}
>  
>  	INIT_LIST_HEAD(&newrec->endp_list);
> @@ -594,6 +739,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
>  		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
>  	else
>  		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
> +	INIT_DELAYED_WORK(&newrec->dev_loss_work, nvme_fc_rport_dev_loss_work);
>  
>  	spin_lock_irqsave(&nvme_fc_lock, flags);
>  	list_add_tail(&newrec->endp_list, &lport->endp_list);
> @@ -604,10 +750,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
>  	*portptr = &newrec->remoteport;
>  	return 0;
>  
> -out_lport_put:
> -	nvme_fc_lport_put(lport);
>  out_kfree_rport:
>  	kfree(newrec);
> +out_lport_put:
> +	nvme_fc_lport_put(lport);
>  out_reghost_failed:
>  	*portptr = NULL;
>  	return ret;
> @@ -638,6 +784,61 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
>  	return 0;
>  }
>  
> +static void
> +nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
> +{
> +	dev_info(ctrl->ctrl.device,
> +		"NVME-FC{%d}: controller connectivity lost. Awaiting "
> +		"Reconnect", ctrl->cnum);
> +
> +	switch (ctrl->ctrl.state) {
> +	case NVME_CTRL_NEW:
> +	case NVME_CTRL_LIVE:
> +		/*
> +		 * Schedule a controller reset. The reset will
> +		 * terminate the association and schedule the
> +		 * reconnect timer. Reconnects will be attempted
> +		 * until either the ctlr_loss_tmo
> +		 * (max_retries * connect_delay) expires or the
> +		 * remoteport's dev_loss_tmo expires.
> +		 */
> +		if (nvme_reset_ctrl(&ctrl->ctrl)) {
> +			dev_warn(ctrl->ctrl.device,
> +				"NVME-FC{%d}: Couldn't schedule reset. "
> +				"Deleting controller.\n",
> +				ctrl->cnum);
> +			__nvme_fc_del_ctrl(ctrl);
> +		}
> +		break;
> +
> +	case NVME_CTRL_RECONNECTING:
> +		/*
> +		 * The association has already been terminated
> +		 * and the controller is attempting reconnects.
> +		 * No need to do anything futher. Reconnects will
> +		 * be attempted until either the ctlr_loss_tmo
> +		 * (max_retries * connect_delay) expires or the
> +		 * remoteport's dev_loss_tmo expires.
> +		 */
> +		break;
> +
> +	case NVME_CTRL_RESETTING:
> +		/*
> +		 * Controller is already in the process of
> +		 * terminating the association. No need to do
> +		 * anything further. The reconnect step will
> +		 * kick in naturally after the association is
> +		 * terminated.
> +		 */
> +		break;
> +
> +	case NVME_CTRL_DELETING:
> +	default:
> +		/* no action to take - let it delete */
> +		break;
> +	}
> +}
> +
>  /**
>   * nvme_fc_unregister_remoteport - transport entry point called by an
>   *                              LLDD to deregister/remove a previously
> @@ -667,15 +868,32 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
>  	}
>  	portptr->port_state = FC_OBJSTATE_DELETED;
>  
> -	/* tear down all associations to the remote port */
> -	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
> -		__nvme_fc_del_ctrl(ctrl);
> +	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
> +		/* if dev_loss_tmo==0, dev loss is immediate */
> +		if (!portptr->dev_loss_tmo) {
> +			dev_warn(ctrl->ctrl.device,
> +				"NVME-FC{%d}: controller connectivity lost. "
> +				"Deleting controller.\n",
> +				ctrl->cnum);
> +			__nvme_fc_del_ctrl(ctrl);
> +		} else
> +			nvme_fc_ctrl_connectivity_loss(ctrl);
> +	}
>  
>  	spin_unlock_irqrestore(&rport->lock, flags);
>  
>  	nvme_fc_abort_lsops(rport);
>  
> +	queue_delayed_work(nvme_wq, &rport->dev_loss_work,
> +			portptr->dev_loss_tmo * HZ);
> +
> +	/*nvme_fc_ctrl_connectivity_loss
> +	 * release the reference, which will allow, if all controllers
> +	 * go away, which should only occur after dev_loss_tmo occurs,
> +	 * for the rport to be torn down.
> +	 */
>  	nvme_fc_rport_put(rport);
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(nvme_fc_unregister_remoteport);

Hmm.
The dev_loss_tmo is pretty awkward; scheduling a per-rport structure
only to tear down all attached _controllers_ once an rport goes down.
But seeing that __nvme_fc_del_ctrl() is actually just punting the
deletion to a workqueue I guess that's okay.
Also we might end up having called __nvme_fc_del_ctrl() for each
controller, making me wonder why we need to schedule a dev_loss_work if
all controllers are being removed anyway.
Also I'm not exactly thrilled with the reference counting; we put things
on a workqueue but do not increase the reference for it.
Which means that the rport structure might be freed even though it's
still on the workqueue. Shouldn't we rather take a reference before
putting it on the workqueue, and drop the reference once we're done with
the workqueue function?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From jthumshirn at suse.de  Fri Oct 20 01:15:46 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 20 Oct 2017 10:15:46 +0200
Subject: [PATCH v2 1/2] nvme_fc: correct io termination handling
In-Reply-To: <20171019231139.23611-2-jsmart2021@gmail.com> (James Smart's
 message of "Thu, 19 Oct 2017 16:11:38 -0700")
References: <20171019231139.23611-1-jsmart2021@gmail.com>
 <20171019231139.23611-2-jsmart2021@gmail.com>
Message-ID: <mqda80mqlot.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Fri Oct 20 01:18:38 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Fri, 20 Oct 2017 10:18:38 +0200
Subject: [PATCH v2 2/2] nvme_fc: correct io timeout behavior
In-Reply-To: <20171019231139.23611-3-jsmart2021@gmail.com> (James Smart's
 message of "Thu, 19 Oct 2017 16:11:39 -0700")
References: <20171019231139.23611-1-jsmart2021@gmail.com>
 <20171019231139.23611-3-jsmart2021@gmail.com>
Message-ID: <mqd4lquqlk1.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From nitzanc at mellanox.com  Fri Oct 20 01:24:22 2017
From: nitzanc at mellanox.com (Nitzan Carmi)
Date: Fri, 20 Oct 2017 08:24:22 +0000
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
Message-ID: <20171020082422.4888-1-nitzanc@mellanox.com>

Signed-off-by: Nitzan Carmi <nitzanc at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/host/rdma.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..8d2bbd7 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1581,6 +1581,12 @@ static enum blk_eh_timer_return
 nvme_rdma_timeout(struct request *rq, bool reserved)
 {
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_command *cmd = req->sqe.data;
+
+	dev_dbg_ratelimited(req->queue->ctrl->ctrl.device,
+			    "Reached timeout: command id %d opcode %d qid %d",
+			    cmd->common.command_id, cmd->common.opcode,
+			    nvme_rdma_queue_idx(req->queue));
 
 	/* queue error recovery */
 	nvme_rdma_error_recovery(req->queue->ctrl);
-- 
2.9.4



From hch at lst.de  Fri Oct 20 03:03:57 2017
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 20 Oct 2017 12:03:57 +0200
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
In-Reply-To: <20171020082422.4888-1-nitzanc@mellanox.com>
References: <20171020082422.4888-1-nitzanc@mellanox.com>
Message-ID: <20171020100356.GA23323@lst.de>

Can you please follow the message format/type used by the PCIe driver?


From hch at infradead.org  Fri Oct 20 03:12:01 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 20 Oct 2017 03:12:01 -0700
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <20171019154821.GD2768@localhost.localdomain>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
 <20171019102847.GA6828@infradead.org>
 <20171019154821.GD2768@localhost.localdomain>
Message-ID: <20171020101201.GA26703@infradead.org>

On Thu, Oct 19, 2017 at 09:48:22AM -0600, Keith Busch wrote:
> Yeah, I don't have any major concerns. I was hung up on the
> tagset.cmd_size being smaller when SGL's are used, but it took me a
> minute to determine that's okay as we were overallocating it before.

How is it smaller?  The code ensures we use the larger of the two
calculations:

	dev->tagset.cmd_size = nvme_pci_cmd_size(dev, false);
	if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
		dev->tagset.cmd_size = max(dev->tagset.cmd_size,
				nvme_pci_cmd_size(dev, true));
	}


From hch at infradead.org  Fri Oct 20 03:12:28 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 20 Oct 2017 03:12:28 -0700
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <20171020101228.GB26703@infradead.org>

Thanks,

applied to nvme-4.15.


From hch at infradead.org  Fri Oct 20 03:13:30 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 20 Oct 2017 03:13:30 -0700
Subject: [PATCH v2 0/2] nvme_fc: rework io timeouts and termination
In-Reply-To: <20171019231139.23611-1-jsmart2021@gmail.com>
References: <20171019231139.23611-1-jsmart2021@gmail.com>
Message-ID: <20171020101330.GC26703@infradead.org>

Thanks,

applied the series to nvme-4.15.


From hch at infradead.org  Fri Oct 20 03:27:21 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 20 Oct 2017 03:27:21 -0700
Subject: [REPOST PATCH] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171019200236.15892-1-jsmart2021@gmail.com>
References: <20171019200236.15892-1-jsmart2021@gmail.com>
Message-ID: <20171020102721.GA20587@infradead.org>

Why can't we just switch to flush_work instead of cancel_work_sync?

fatal_err_work never requeues itself, although we might need
protection against queueing more up once we enter nvmet_ctrl_free.


From hch at infradead.org  Fri Oct 20 06:17:39 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 20 Oct 2017 15:17:39 +0200
Subject: [GIT PULL] nvme fixes for 4.14
Message-ID: <20171020131739.nffgzny5465tboba@infradead.org>

Hi Jens,

below are two regression fixes each for RDMA and FC, and a fix for a SQHD
update race in the target.

The following changes since commit 639812a1ed9bf49ae2c026086fbf975339cd1eef:

  nbd: don't set the device size until we're connected (2017-10-09 12:29:22 -0600)

are available in the git repository at:

  git://git.infradead.org/nvme.git nvme-4.14

for you to fetch changes up to f04b9cc87b5fc466b1b7231ba7b078e885956c5b:

  nvme-rdma: Fix error status return in tagset allocation failure (2017-10-19 17:13:51 +0200)

----------------------------------------------------------------
James Smart (3):
      nvme-fc: fix iowait hang
      nvme-fc: retry initial controller connections 3 times
      nvmet: synchronize sqhd update

Sagi Grimberg (2):
      nvme-rdma: Fix possible double free in reconnect flow
      nvme-rdma: Fix error status return in tagset allocation failure

 drivers/nvme/host/fc.c      | 37 +++++++++++++++++++++++++++++++++----
 drivers/nvme/host/rdma.c    | 16 ++++++++++++----
 drivers/nvme/target/core.c  | 15 ++++++++++++---
 drivers/nvme/target/nvmet.h |  2 +-
 4 files changed, 58 insertions(+), 12 deletions(-)


From axboe at kernel.dk  Fri Oct 20 07:04:42 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Fri, 20 Oct 2017 09:04:42 -0500
Subject: [GIT PULL] nvme fixes for 4.14
In-Reply-To: <20171020131739.nffgzny5465tboba@infradead.org>
References: <20171020131739.nffgzny5465tboba@infradead.org>
Message-ID: <ca329eec-ce4c-5899-4a32-419859196a22@kernel.dk>

On 10/20/2017 08:17 AM, Christoph Hellwig wrote:
> Hi Jens,
> 
> below are two regression fixes each for RDMA and FC, and a fix for a SQHD
> update race in the target.
> 
> The following changes since commit 639812a1ed9bf49ae2c026086fbf975339cd1eef:
> 
>   nbd: don't set the device size until we're connected (2017-10-09 12:29:22 -0600)
> 
> are available in the git repository at:
> 
>   git://git.infradead.org/nvme.git nvme-4.14

Pulled, thanks.

-- 
Jens Axboe



From hare at suse.de  Fri Oct 20 07:11:15 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 20 Oct 2017 16:11:15 +0200
Subject: [REPOST PATCH] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171020102721.GA20587@infradead.org>
References: <20171019200236.15892-1-jsmart2021@gmail.com>
 <20171020102721.GA20587@infradead.org>
Message-ID: <cde5ef37-4a23-5257-c76a-425ece92b777@suse.de>

On 10/20/2017 12:27 PM, Christoph Hellwig wrote:
> Why can't we just switch to flush_work instead of cancel_work_sync?
> 
> fatal_err_work never requeues itself, although we might need
> protection against queueing more up once we enter nvmet_ctrl_free.
> 
That's actually what I would prefer, too.

Deadlocking when doing to final put on the controller from the
fatal_error queue function is not a nice design.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From keith.busch at intel.com  Fri Oct 20 07:37:00 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 08:37:00 -0600
Subject: [PATCH V7] nvme-pci: add SGL support
In-Reply-To: <20171020101201.GA26703@infradead.org>
References: <1508203460-13852-1-git-send-email-ckulkarnilinux@gmail.com>
 <20171019102847.GA6828@infradead.org>
 <20171019154821.GD2768@localhost.localdomain>
 <20171020101201.GA26703@infradead.org>
Message-ID: <20171020143659.GA3493@localhost.localdomain>

On Fri, Oct 20, 2017 at 03:12:01AM -0700, Christoph Hellwig wrote:
> On Thu, Oct 19, 2017 at 09:48:22AM -0600, Keith Busch wrote:
> > Yeah, I don't have any major concerns. I was hung up on the
> > tagset.cmd_size being smaller when SGL's are used, but it took me a
> > minute to determine that's okay as we were overallocating it before.
> 
> How is it smaller?  The code ensures we use the larger of the two
> calculations:
> 
> 	dev->tagset.cmd_size = nvme_pci_cmd_size(dev, false);
> 	if ((dev->ctrl.sgls & ((1 << 0) | (1 << 1))) && sgl_threshold) {
> 		dev->tagset.cmd_size = max(dev->tagset.cmd_size,
> 				nvme_pci_cmd_size(dev, true));
> 	}

Oops, my mistake.


From jsmart2021 at gmail.com  Fri Oct 20 07:50:45 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 07:50:45 -0700
Subject: [PATCH v3 3/5] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <ee0d6875-843f-103e-fa1e-825bf18acf48@suse.de>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-4-jsmart2021@gmail.com>
 <ee0d6875-843f-103e-fa1e-825bf18acf48@suse.de>
Message-ID: <390ba07b-bdd5-ec3c-ded2-dd1286404025@gmail.com>

On 10/19/2017 11:04 PM, Hannes Reinecke wrote:
> Is this exported to sysfs somewhere? If not, can we have it in sysfs?
> It surely made live _so_ much easier when configuring the odd device...
> 
> Other than that:
> 
> Reviewed-by: Hannes Reinecke <hare at suse.com>
> 
> Cheers,
> 
> Hannes
> 

Currently, the scsi transpoirt fc_rport sysfs entry has the dev_loss_tmo 
attribute, and when set, the driver will call the 
nvme_fc_set_remoteport_devloss() routine to set it on the nvme-fc 
remoteport.

When we genericize the fc transport, it will be on the generic fc_rport, 
and the set will call both the SCSI transport/lldd and the NVME remoteport.

-- james


From jsmart2021 at gmail.com  Fri Oct 20 09:12:53 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 09:12:53 -0700
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
Message-ID: <20171020161253.29296-1-jsmart2021@gmail.com>

Below is  a stack trace for an issue that was reported.

What's happening is that the nvmet layer had it's controller kato
timeout fire, which causes it to schedule its fatal error handler
via the fatal_err_work element. The error handler is invoked, which
calls the transport delete_ctrl() entry point, and as the transport
tears down the controller, nvmet_sq_destroy ends up doing the final
put on the ctlr causing it to enter its free routine. The ctlr free
routine does a cancel_work_sync() on fatal_err_work element, which
then does a flush_work and wait_for_completion. But, as the wait is
in the context of the work element being flushed, its in a catch-22
and the thread hangs.

[  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  326.909832] nvmet: ctrl 1 fatal error occurred!
[  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
x114 flg x2
[  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
seconds.
[  494.589638]       Not tainted 4.14.0-rc1.James+ #1
[  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  494.603718] kworker/0:2     D    0   243      2 0x80000000
[  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
[  494.616447] Call Trace:
[  494.619177]  __schedule+0x28d/0x890
[  494.623070]  schedule+0x36/0x80
[  494.626571]  schedule_timeout+0x1dd/0x300
[  494.631044]  ? dequeue_task_fair+0x592/0x840
[  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
[  494.640756]  wait_for_completion+0x121/0x180
[  494.645521]  ? wake_up_q+0x80/0x80
[  494.649315]  flush_work+0x11d/0x1a0
[  494.653206]  ? wake_up_worker+0x30/0x30
[  494.657484]  __cancel_work_timer+0x10b/0x190
[  494.662249]  cancel_work_sync+0x10/0x20
[  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
[  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
[  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
[  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
[  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
[  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
[  494.701589]  process_one_work+0x149/0x360
[  494.706064]  worker_thread+0x4d/0x3c0
[  494.710148]  kthread+0x109/0x140
[  494.713751]  ? rescuer_thread+0x380/0x380
[  494.718214]  ? kthread_park+0x60/0x60

Correct by creating a final free work element, which is scheduled
by the final ctrl put routine, so that when the flush_work (was
cancel_work_sync) is called, it cannot be in the same work_q
element context.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2: convert cancel_work_sync() to flush_work() for fatal_err_work
---
 drivers/nvme/target/core.c  | 28 +++++++++++++++++++---------
 drivers/nvme/target/nvmet.h |  1 +
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 1b208beeef50..3bc671abdeb7 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -840,19 +840,14 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	return status;
 }
 
-static void nvmet_ctrl_free(struct kref *ref)
+static void nvmet_ctrl_free_work(struct work_struct *work)
 {
-	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_ctrl *ctrl =
+			container_of(work, struct nvmet_ctrl, free_work);
 	struct nvmet_subsys *subsys = ctrl->subsys;
 
-	nvmet_stop_keep_alive_timer(ctrl);
-
-	mutex_lock(&subsys->lock);
-	list_del(&ctrl->subsys_entry);
-	mutex_unlock(&subsys->lock);
-
 	flush_work(&ctrl->async_event_work);
-	cancel_work_sync(&ctrl->fatal_err_work);
+	flush_work(&ctrl->fatal_err_work);
 
 	ida_simple_remove(&cntlid_ida, ctrl->cntlid);
 	nvmet_subsys_put(subsys);
@@ -862,6 +857,21 @@ static void nvmet_ctrl_free(struct kref *ref)
 	kfree(ctrl);
 }
 
+static void nvmet_ctrl_free(struct kref *ref)
+{
+	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_subsys *subsys = ctrl->subsys;
+
+	nvmet_stop_keep_alive_timer(ctrl);
+
+	mutex_lock(&subsys->lock);
+	list_del(&ctrl->subsys_entry);
+	mutex_unlock(&subsys->lock);
+
+	INIT_WORK(&ctrl->free_work, nvmet_ctrl_free_work);
+	schedule_work(&ctrl->free_work);
+}
+
 void nvmet_ctrl_put(struct nvmet_ctrl *ctrl)
 {
 	kref_put(&ctrl->ref, nvmet_ctrl_free);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index e342f02845c1..ed38b44a7007 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -129,6 +129,7 @@ struct nvmet_ctrl {
 	struct kref		ref;
 	struct delayed_work	ka_work;
 	struct work_struct	fatal_err_work;
+	struct work_struct	free_work;
 
 	struct nvmet_fabrics_ops *ops;
 
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 11:53:07 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 11:53:07 -0700
Subject: [PATCH v3 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <1ea7f564-36ed-a952-22c3-4f7b58be6665@suse.de>
References: <20171017233248.6769-1-jsmart2021@gmail.com>
 <20171017233248.6769-6-jsmart2021@gmail.com>
 <1ea7f564-36ed-a952-22c3-4f7b58be6665@suse.de>
Message-ID: <7cd69378-9ce8-6734-f164-9cd1b72b53eb@gmail.com>

On 10/19/2017 11:27 PM, Hannes Reinecke wrote:
...
>> +static struct nvme_fc_rport *
>> +nvme_fc_attach_to_suspended_rport(struct nvme_fc_lport *lport,
>> +				struct nvme_fc_port_info *pinfo)
>> +{
...
>> +
>> +		/* has it been unregistered */
>> +		if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
>> +			/* means lldd called us twice */
>> +			spin_unlock_irqrestore(&rport->lock, flags);
>> +			nvme_fc_rport_put(rport);
>> +			return ERR_PTR(-ESTALE);
>> +		}
>> +
> Why can't we re-use this rport?
> It's patently _not_ deleted, so we should be good to use it, right?
> Someone else might've messed up, but if _we_ get our reference counting
> right that shouldn't affect us, no?
> Plus: How would we recover from such a situation?
> The rport most likely will _never_ go away, meaning we'll always hit
> this scenario for each and every connection attempt.
> (Speaking from experience here; I've hit this scenario accidentally :-)
> So the only chance we have is a reboot.


Because if it's not in a deleted state, then it's one that is 
effectively live/present and there can be only 1 remoteport with the 
same WWPN/WWNN on the localport.  This would be the case of a lldd 
calling register for the same remoteport twice, which we don't allow.
I'm not looking to allow or survive such a driver error - I'll let the 
second registration fail as it is illegal.

How did you hit this scenario ? it would really surprise me.



>>   }
>>   
>> +static void
>> +nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
>> +{
>> +	dev_info(ctrl->ctrl.device,
>> +		"NVME-FC{%d}: controller connectivity lost. Awaiting "
>> +		"Reconnect", ctrl->cnum);
>> +
>> +	switch (ctrl->ctrl.state) {
>> +	case NVME_CTRL_NEW:
>> +	case NVME_CTRL_LIVE:
>> +		/*
>> +		 * Schedule a controller reset. The reset will
>> +		 * terminate the association and schedule the
>> +		 * reconnect timer. Reconnects will be attempted
>> +		 * until either the ctlr_loss_tmo
>> +		 * (max_retries * connect_delay) expires or the
>> +		 * remoteport's dev_loss_tmo expires.
>> +		 */
>> +		if (nvme_reset_ctrl(&ctrl->ctrl)) {
>> +			dev_warn(ctrl->ctrl.device,
>> +				"NVME-FC{%d}: Couldn't schedule reset. "
>> +				"Deleting controller.\n",
>> +				ctrl->cnum);
>> +			__nvme_fc_del_ctrl(ctrl);
>> +		}
>> +		break;
>> +
>> +	case NVME_CTRL_RECONNECTING:
>> +		/*
>> +		 * The association has already been terminated
>> +		 * and the controller is attempting reconnects.
>> +		 * No need to do anything futher. Reconnects will
>> +		 * be attempted until either the ctlr_loss_tmo
>> +		 * (max_retries * connect_delay) expires or the
>> +		 * remoteport's dev_loss_tmo expires.
>> +		 */
>> +		break;
>> +
>> +	case NVME_CTRL_RESETTING:
>> +		/*
>> +		 * Controller is already in the process of
>> +		 * terminating the association. No need to do
>> +		 * anything further. The reconnect step will
>> +		 * kick in naturally after the association is
>> +		 * terminated.
>> +		 */
>> +		break;
>> +
>> +	case NVME_CTRL_DELETING:
>> +	default:
>> +		/* no action to take - let it delete */
>> +		break;
>> +	}
>> +}
>> +
>>   /**
>>    * nvme_fc_unregister_remoteport - transport entry point called by an
>>    *                              LLDD to deregister/remove a previously
>> @@ -667,15 +868,32 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
>>   	}
>>   	portptr->port_state = FC_OBJSTATE_DELETED;
>>   
>> -	/* tear down all associations to the remote port */
>> -	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
>> -		__nvme_fc_del_ctrl(ctrl);
>> +	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
>> +		/* if dev_loss_tmo==0, dev loss is immediate */
>> +		if (!portptr->dev_loss_tmo) {
>> +			dev_warn(ctrl->ctrl.device,
>> +				"NVME-FC{%d}: controller connectivity lost. "
>> +				"Deleting controller.\n",
>> +				ctrl->cnum);
>> +			__nvme_fc_del_ctrl(ctrl);
>> +		} else
>> +			nvme_fc_ctrl_connectivity_loss(ctrl);
>> +	}
>>   
>>   	spin_unlock_irqrestore(&rport->lock, flags);
>>   
>>   	nvme_fc_abort_lsops(rport);
>>   
>> +	queue_delayed_work(nvme_wq, &rport->dev_loss_work,
>> +			portptr->dev_loss_tmo * HZ);
>> +
>> +	/*nvme_fc_ctrl_connectivity_loss
>> +	 * release the reference, which will allow, if all controllers
>> +	 * go away, which should only occur after dev_loss_tmo occurs,
>> +	 * for the rport to be torn down.
>> +	 */
>>   	nvme_fc_rport_put(rport);
>> +
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL_GPL(nvme_fc_unregister_remoteport);
> 
> Hmm.
> The dev_loss_tmo is pretty awkward; scheduling a per-rport structure
> only to tear down all attached _controllers_ once an rport goes down.
> But seeing that __nvme_fc_del_ctrl() is actually just punting the
> deletion to a workqueue I guess that's okay.

I guess so. Just reusing the existing delete code paths whcih set the
nvme controller state.

> Also we might end up having called __nvme_fc_del_ctrl() for each
> controller, making me wonder why we need to schedule a dev_loss_work if
> all controllers are being removed anyway.

I'm not sure I follow, but..  We could defer to the ctlr_loss_tmo 
completely, but it may be longer than dev_loss_tmo and I think the 
confuses people that may set the dev_loss_tmo value an expect things to 
terminate at dev_loss_tmo. Currently, the controller delete is 
min(dev_loss_tmo, ctlr_loss_tmo).


> Also I'm not exactly thrilled with the reference counting; we put things
> on a workqueue but do not increase the reference for it.
> Which means that the rport structure might be freed even though it's
> still on the workqueue. Shouldn't we rather take a reference before
> putting it on the workqueue, and drop the reference once we're done with
> the workqueue function?

ok, I agree.  I'll rework so that there is no dev_loss_tmo timer 
running. Instead, we'll record the rport connectivity loss time, and in 
the controller reconnect loops, if we try to reconnect but connectivity 
has been lost for dev_loss_tmo, we'll give up.

-- james



From jg at lightnvm.io  Fri Oct 20 12:03:46 2017
From: jg at lightnvm.io (=?utf-8?Q?Javier_Gonz=C3=A1lez?=)
Date: Fri, 20 Oct 2017 21:03:46 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171018165258.23212-16-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
 <20171018165258.23212-16-hch@lst.de>
Message-ID: <13C5EC7C-6E04-4D10-8F6D-B816BB39CF9A@lightnvm.io>

Javier
> On 18 Oct 2017, at 18.52, Christoph Hellwig <hch at lst.de> wrote:
> 
> Introduce a new struct nvme_ns_head that holds information about an actual
> namespace, unlike struct nvme_ns, which only holds the per-controller
> namespace information.  For private namespaces there is a 1:1 relation of
> the two, but for shared namespaces this lets us discover all the paths to
> it.  For now only the identifiers are moved to the new structure, but most
> of the information in struct nvme_ns should eventually move over.
> 
> To allow lockless path lookup the list of nvme_ns structures per
> nvme_ns_head is protected by SRCU, which requires freeing the nvme_ns
> structure through call_srcu.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Keith Busch <keith.busch at intel.com>
> ---
> drivers/nvme/host/core.c     | 192 +++++++++++++++++++++++++++++++++++++------
> drivers/nvme/host/lightnvm.c |  14 ++--
> drivers/nvme/host/nvme.h     |  21 ++++-
> 3 files changed, 192 insertions(+), 35 deletions(-)
> 
> 
LGTM

Reviewed-by: Javier Gonz?lez <javier at cnexlabs.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20171020/73f9b2c1/attachment.sig>

From keith.busch at intel.com  Fri Oct 20 15:18:03 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:18:03 -0600
Subject: [PATCH] nvme: Remove unused headers
Message-ID: <20171020221803.30484-1-keith.busch@intel.com>

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/pci.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 11e6fd9d0ba4..7e4d704ec198 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -13,7 +13,6 @@
  */
 
 #include <linux/aer.h>
-#include <linux/bitops.h>
 #include <linux/blkdev.h>
 #include <linux/blk-mq.h>
 #include <linux/blk-mq-pci.h>
@@ -26,12 +25,9 @@
 #include <linux/mutex.h>
 #include <linux/once.h>
 #include <linux/pci.h>
-#include <linux/poison.h>
 #include <linux/t10-pi.h>
-#include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
-#include <asm/unaligned.h>
 #include <linux/sed-opal.h>
 
 #include "nvme.h"
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:19 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:19 -0600
Subject: [PATCHv2 0/5] AEN and userspace updates
Message-ID: <20171020221924.30659-1-keith.busch@intel.com>

This series cleans up some duplicate AEN code, and ultimately adds change
uevents for nvme devices that raise an AEN that the driver did not handle.

I have updated nvme-cli to have a more convenient way to retreive a log
based on the AEN result, and intend to provide a udev rule to wrap that
command when this goes through.

v1 -> v2;

  Merge with nvme-4.15.

  Squashed the AEN define centralization patches.

  Using kasprintf for the uevent.

  Removed async event helper function and made it inline.

  Filter AEN notification only for events the kernel does not handle.
  This would be useful to prevent userspace from reacting events and
  reading logs that may alter what the kernel sees.

Keith Busch (5):
  nvme: Centralize AEN defines
  nvme/fc: remove unused "queue_size" field
  nvme: Single AEN request
  nvme: Unexport starting async event work
  nvme: Send uevent for unhandled AEN completions

 drivers/nvme/host/core.c   | 57 ++++++++++++++++++++--------------------------
 drivers/nvme/host/fc.c     | 47 +++++++++++++-------------------------
 drivers/nvme/host/nvme.h   |  6 ++---
 drivers/nvme/host/pci.c    | 14 ++++--------
 drivers/nvme/host/rdma.c   | 19 ++++------------
 drivers/nvme/target/loop.c | 16 ++++---------
 include/linux/nvme.h       |  2 ++
 7 files changed, 57 insertions(+), 104 deletions(-)

-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:21 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:21 -0600
Subject: [PATCHv2 2/5] nvme/fc: remove unused "queue_size" field
In-Reply-To: <20171020221924.30659-1-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
Message-ID: <20171020221924.30659-3-keith.busch@intel.com>

This was being saved in a structure, but never used anywhere. The queue
size is obtained through other means, so there's no reason to duplicate
this without a user for it.

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/fc.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index e91e81d9f9fd..6b8e6ec3c2ab 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -41,7 +41,6 @@ struct nvme_fc_queue {
 	struct device		*dev;
 	struct blk_mq_hw_ctx	*hctx;
 	void			*lldd_handle;
-	int			queue_size;
 	size_t			cmnd_capsule_len;
 	u32			qnum;
 	u32			rqcnt;
@@ -1662,7 +1661,7 @@ nvme_fc_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 }
 
 static void
-nvme_fc_init_queue(struct nvme_fc_ctrl *ctrl, int idx, size_t queue_size)
+nvme_fc_init_queue(struct nvme_fc_ctrl *ctrl, int idx)
 {
 	struct nvme_fc_queue *queue;
 
@@ -1678,8 +1677,6 @@ nvme_fc_init_queue(struct nvme_fc_ctrl *ctrl, int idx, size_t queue_size)
 	else
 		queue->cmnd_capsule_len = sizeof(struct nvme_command);
 
-	queue->queue_size = queue_size;
-
 	/*
 	 * Considered whether we should allocate buffers for all SQEs
 	 * and CQEs and dma map them - mapping their respective entries
@@ -1803,7 +1800,7 @@ nvme_fc_init_io_queues(struct nvme_fc_ctrl *ctrl)
 	int i;
 
 	for (i = 1; i < ctrl->ctrl.queue_count; i++)
-		nvme_fc_init_queue(ctrl, i, ctrl->ctrl.sqsize);
+		nvme_fc_init_queue(ctrl, i);
 }
 
 static void
@@ -2436,7 +2433,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	 * Create the admin queue
 	 */
 
-	nvme_fc_init_queue(ctrl, 0, NVME_AQ_BLK_MQ_DEPTH);
+	nvme_fc_init_queue(ctrl, 0);
 
 	ret = __nvme_fc_create_hw_queue(ctrl, &ctrl->queues[0], 0,
 				NVME_AQ_BLK_MQ_DEPTH);
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:22 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:22 -0600
Subject: [PATCHv2 3/5] nvme: Single AEN request
In-Reply-To: <20171020221924.30659-1-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
Message-ID: <20171020221924.30659-4-keith.busch@intel.com>

The driver can handle tracking only one AEN request, so this patch
removes handling for multiple ones.

Reviewed-by: James Smart  <james.smart at broadcom.com>
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c   | 28 +++-------------------------
 drivers/nvme/host/fc.c     |  9 +++------
 drivers/nvme/host/nvme.h   |  3 +--
 drivers/nvme/host/pci.c    |  4 ++--
 drivers/nvme/host/rdma.c   |  5 +----
 drivers/nvme/target/loop.c |  2 +-
 6 files changed, 11 insertions(+), 40 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2b7fa9ba1e22..7fb55b6e26c3 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2579,15 +2579,7 @@ static void nvme_async_event_work(struct work_struct *work)
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, async_event_work);
 
-	spin_lock_irq(&ctrl->lock);
-	while (ctrl->state == NVME_CTRL_LIVE && ctrl->event_limit > 0) {
-		int aer_idx = --ctrl->event_limit;
-
-		spin_unlock_irq(&ctrl->lock);
-		ctrl->ops->submit_async_event(ctrl, aer_idx);
-		spin_lock_irq(&ctrl->lock);
-	}
-	spin_unlock_irq(&ctrl->lock);
+	ctrl->ops->submit_async_event(ctrl);
 }
 
 static bool nvme_ctrl_pp_status(struct nvme_ctrl *ctrl)
@@ -2659,22 +2651,8 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		union nvme_result *res)
 {
 	u32 result = le32_to_cpu(res->u32);
-	bool done = true;
 
-	switch (le16_to_cpu(status) >> 1) {
-	case NVME_SC_SUCCESS:
-		done = false;
-		/*FALLTHRU*/
-	case NVME_SC_ABORT_REQ:
-		++ctrl->event_limit;
-		if (ctrl->state == NVME_CTRL_LIVE)
-			queue_work(nvme_wq, &ctrl->async_event_work);
-		break;
-	default:
-		break;
-	}
-
-	if (done)
+	if (le16_to_cpu(status) >> 1 != NVME_SC_SUCCESS)
 		return;
 
 	switch (result & 0xff07) {
@@ -2688,12 +2666,12 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 	default:
 		dev_warn(ctrl->device, "async event result %08x\n", result);
 	}
+	queue_work(nvme_wq, &ctrl->async_event_work);
 }
 EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 
 void nvme_queue_async_events(struct nvme_ctrl *ctrl)
 {
-	ctrl->event_limit = NVME_AEN_COMMANDS;
 	queue_work(nvme_wq, &ctrl->async_event_work);
 }
 EXPORT_SYMBOL_GPL(nvme_queue_async_events);
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 6b8e6ec3c2ab..285e0579bbef 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2165,7 +2165,7 @@ nvme_fc_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
 }
 
 static void
-nvme_fc_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
+nvme_fc_submit_async_event(struct nvme_ctrl *arg)
 {
 	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(arg);
 	struct nvme_fc_fcp_op *aen_op;
@@ -2173,9 +2173,6 @@ nvme_fc_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	bool terminating = false;
 	blk_status_t ret;
 
-	if (aer_idx > NVME_AEN_COMMANDS)
-		return;
-
 	spin_lock_irqsave(&ctrl->lock, flags);
 	if (ctrl->flags & FCCTRL_TERMIO)
 		terminating = true;
@@ -2184,13 +2181,13 @@ nvme_fc_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	if (terminating)
 		return;
 
-	aen_op = &ctrl->aen_ops[aer_idx];
+	aen_op = &ctrl->aen_ops[0];
 
 	ret = nvme_fc_start_fcp_op(ctrl, aen_op->queue, aen_op, 0,
 					NVMEFC_FCP_NODATA);
 	if (ret)
 		dev_err(ctrl->ctrl.device,
-			"failed async event work [%d]\n", aer_idx);
+			"failed async event work\n");
 }
 
 static void
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 331d91d5e191..2f1e52131430 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -160,7 +160,6 @@ struct nvme_ctrl {
 	u16 nssa;
 	u16 nr_streams;
 	atomic_t abort_limit;
-	u8 event_limit;
 	u8 vwc;
 	u32 vs;
 	u32 sgls;
@@ -234,7 +233,7 @@ struct nvme_ctrl_ops {
 	int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
 	int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 	void (*free_ctrl)(struct nvme_ctrl *ctrl);
-	void (*submit_async_event)(struct nvme_ctrl *ctrl, int aer_idx);
+	void (*submit_async_event)(struct nvme_ctrl *ctrl);
 	int (*delete_ctrl)(struct nvme_ctrl *ctrl);
 	int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
 	int (*reinit_request)(void *data, struct request *rq);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 52422c3a106f..73cb97e2121e 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1043,7 +1043,7 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
 	return __nvme_poll(nvmeq, tag);
 }
 
-static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl, int aer_idx)
+static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl)
 {
 	struct nvme_dev *dev = to_nvme_dev(ctrl);
 	struct nvme_queue *nvmeq = dev->queues[0];
@@ -1051,7 +1051,7 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl, int aer_idx)
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = nvme_admin_async_event;
-	c.common.command_id = NVME_AQ_BLK_MQ_DEPTH + aer_idx;
+	c.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
 
 	spin_lock_irq(&nvmeq->q_lock);
 	__nvme_submit_cmd(nvmeq, &c);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index f35992b1c48b..9b610ba22eb9 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1294,7 +1294,7 @@ static struct blk_mq_tags *nvme_rdma_tagset(struct nvme_rdma_queue *queue)
 	return queue->ctrl->tag_set.tags[queue_idx - 1];
 }
 
-static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
+static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg)
 {
 	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(arg);
 	struct nvme_rdma_queue *queue = &ctrl->queues[0];
@@ -1304,9 +1304,6 @@ static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	struct ib_sge sge;
 	int ret;
 
-	if (WARN_ON_ONCE(aer_idx != 0))
-		return;
-
 	ib_dma_sync_single_for_cpu(dev, sqe->dma, sizeof(*cmd), DMA_TO_DEVICE);
 
 	memset(cmd, 0, sizeof(*cmd));
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index fc2856943d89..2b6c75d66def 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -185,7 +185,7 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
-static void nvme_loop_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
+static void nvme_loop_submit_async_event(struct nvme_ctrl *arg)
 {
 	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(arg);
 	struct nvme_loop_queue *queue = &ctrl->queues[0];
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:20 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:20 -0600
Subject: [PATCHv2 1/5] nvme: Centralize AEN defines
In-Reply-To: <20171020221924.30659-1-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
Message-ID: <20171020221924.30659-2-keith.busch@intel.com>

All the transports were unnecessarilly duplicating the AEN request
accounting. This defines everything in one place.

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c   |  2 +-
 drivers/nvme/host/fc.c     | 33 ++++++++++++---------------------
 drivers/nvme/host/nvme.h   |  1 -
 drivers/nvme/host/pci.c    | 12 +++---------
 drivers/nvme/host/rdma.c   | 14 +++-----------
 drivers/nvme/target/loop.c | 14 +++-----------
 include/linux/nvme.h       |  2 ++
 7 files changed, 24 insertions(+), 54 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7fae42d595d5..2b7fa9ba1e22 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2693,7 +2693,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 
 void nvme_queue_async_events(struct nvme_ctrl *ctrl)
 {
-	ctrl->event_limit = NVME_NR_AERS;
+	ctrl->event_limit = NVME_AEN_COMMANDS;
 	queue_work(nvme_wq, &ctrl->async_event_work);
 }
 EXPORT_SYMBOL_GPL(nvme_queue_async_events);
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index c6c903f1b172..e91e81d9f9fd 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -30,15 +30,6 @@
 /* *************************** Data Structures/Defines ****************** */
 
 
-/*
- * We handle AEN commands ourselves and don't even let the
- * block layer know about them.
- */
-#define NVME_FC_NR_AEN_COMMANDS	1
-#define NVME_FC_AQ_BLKMQ_DEPTH	\
-	(NVME_AQ_DEPTH - NVME_FC_NR_AEN_COMMANDS)
-#define AEN_CMDID_BASE		(NVME_FC_AQ_BLKMQ_DEPTH + 1)
-
 enum nvme_fc_queue_flags {
 	NVME_FC_Q_CONNECTED = (1 << 0),
 };
@@ -165,7 +156,7 @@ struct nvme_fc_ctrl {
 	u32			iocnt;
 	wait_queue_head_t	ioabort_wait;
 
-	struct nvme_fc_fcp_op	aen_ops[NVME_FC_NR_AEN_COMMANDS];
+	struct nvme_fc_fcp_op	aen_ops[NVME_AEN_COMMANDS];
 
 	struct nvme_ctrl	ctrl;
 };
@@ -1322,7 +1313,7 @@ nvme_fc_abort_aen_ops(struct nvme_fc_ctrl *ctrl)
 	unsigned long flags;
 	int i, ret;
 
-	for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
+	for (i = 0; i < NVME_AEN_COMMANDS; i++, aen_op++) {
 		if (atomic_read(&aen_op->state) != FCPOP_STATE_ACTIVE)
 			continue;
 
@@ -1592,7 +1583,7 @@ nvme_fc_init_aen_ops(struct nvme_fc_ctrl *ctrl)
 	int i, ret;
 
 	aen_op = ctrl->aen_ops;
-	for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
+	for (i = 0; i < NVME_AEN_COMMANDS; i++, aen_op++) {
 		private = kzalloc(ctrl->lport->ops->fcprqst_priv_sz,
 						GFP_KERNEL);
 		if (!private)
@@ -1602,7 +1593,7 @@ nvme_fc_init_aen_ops(struct nvme_fc_ctrl *ctrl)
 		sqe = &cmdiu->sqe;
 		ret = __nvme_fc_init_request(ctrl, &ctrl->queues[0],
 				aen_op, (struct request *)NULL,
-				(AEN_CMDID_BASE + i));
+				(NVME_AQ_BLK_MQ_DEPTH + i));
 		if (ret) {
 			kfree(private);
 			return ret;
@@ -1615,7 +1606,7 @@ nvme_fc_init_aen_ops(struct nvme_fc_ctrl *ctrl)
 		memset(sqe, 0, sizeof(*sqe));
 		sqe->common.opcode = nvme_admin_async_event;
 		/* Note: core layer may overwrite the sqe.command_id value */
-		sqe->common.command_id = AEN_CMDID_BASE + i;
+		sqe->common.command_id = NVME_AQ_BLK_MQ_DEPTH + i;
 	}
 	return 0;
 }
@@ -1627,7 +1618,7 @@ nvme_fc_term_aen_ops(struct nvme_fc_ctrl *ctrl)
 	int i;
 
 	aen_op = ctrl->aen_ops;
-	for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
+	for (i = 0; i < NVME_AEN_COMMANDS; i++, aen_op++) {
 		if (!aen_op->fcp_req.private)
 			continue;
 
@@ -2185,7 +2176,7 @@ nvme_fc_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	bool terminating = false;
 	blk_status_t ret;
 
-	if (aer_idx > NVME_FC_NR_AEN_COMMANDS)
+	if (aer_idx > NVME_AEN_COMMANDS)
 		return;
 
 	spin_lock_irqsave(&ctrl->lock, flags);
@@ -2445,16 +2436,16 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	 * Create the admin queue
 	 */
 
-	nvme_fc_init_queue(ctrl, 0, NVME_FC_AQ_BLKMQ_DEPTH);
+	nvme_fc_init_queue(ctrl, 0, NVME_AQ_BLK_MQ_DEPTH);
 
 	ret = __nvme_fc_create_hw_queue(ctrl, &ctrl->queues[0], 0,
-				NVME_FC_AQ_BLKMQ_DEPTH);
+				NVME_AQ_BLK_MQ_DEPTH);
 	if (ret)
 		goto out_free_queue;
 
 	ret = nvme_fc_connect_admin_queue(ctrl, &ctrl->queues[0],
-				NVME_FC_AQ_BLKMQ_DEPTH,
-				(NVME_FC_AQ_BLKMQ_DEPTH / 4));
+				NVME_AQ_BLK_MQ_DEPTH,
+				(NVME_AQ_BLK_MQ_DEPTH / 4));
 	if (ret)
 		goto out_delete_hw_queue;
 
@@ -2854,7 +2845,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 
 	memset(&ctrl->admin_tag_set, 0, sizeof(ctrl->admin_tag_set));
 	ctrl->admin_tag_set.ops = &nvme_fc_admin_mq_ops;
-	ctrl->admin_tag_set.queue_depth = NVME_FC_AQ_BLKMQ_DEPTH;
+	ctrl->admin_tag_set.queue_depth = NVME_AQ_BLK_MQ_DEPTH - 1;
 	ctrl->admin_tag_set.reserved_tags = 2; /* fabric connect + Keep-Alive */
 	ctrl->admin_tag_set.numa_node = NUMA_NO_NODE;
 	ctrl->admin_tag_set.cmd_size = sizeof(struct nvme_fc_fcp_op) +
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index cb9d93048f3d..331d91d5e191 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -300,7 +300,6 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl);
 int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 		bool send);
 
-#define NVME_NR_AERS	1
 void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		union nvme_result *res);
 void nvme_queue_async_events(struct nvme_ctrl *ctrl);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 7e4d704ec198..52422c3a106f 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -35,12 +35,6 @@
 #define SQ_SIZE(depth)		(depth * sizeof(struct nvme_command))
 #define CQ_SIZE(depth)		(depth * sizeof(struct nvme_completion))
 
-/*
- * We handle AEN commands ourselves and don't even let the
- * block layer know about them.
- */
-#define NVME_AQ_BLKMQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AERS)
-
 #define SGES_PER_PAGE	(PAGE_SIZE / sizeof(struct nvme_sgl_desc))
 
 static int use_threaded_interrupts;
@@ -956,7 +950,7 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,
 	 * for them but rather special case them here.
 	 */
 	if (unlikely(nvmeq->qid == 0 &&
-			cqe->command_id >= NVME_AQ_BLKMQ_DEPTH)) {
+			cqe->command_id >= NVME_AQ_BLK_MQ_DEPTH)) {
 		nvme_complete_async_event(&nvmeq->dev->ctrl,
 				cqe->status, &cqe->result);
 		return;
@@ -1057,7 +1051,7 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl, int aer_idx)
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = nvme_admin_async_event;
-	c.common.command_id = NVME_AQ_BLKMQ_DEPTH + aer_idx;
+	c.common.command_id = NVME_AQ_BLK_MQ_DEPTH + aer_idx;
 
 	spin_lock_irq(&nvmeq->q_lock);
 	__nvme_submit_cmd(nvmeq, &c);
@@ -1528,7 +1522,7 @@ static int nvme_alloc_admin_tags(struct nvme_dev *dev)
 		 * Subtract one to leave an empty queue entry for 'Full Queue'
 		 * condition. See NVM-Express 1.2 specification, section 4.1.2.
 		 */
-		dev->admin_tagset.queue_depth = NVME_AQ_BLKMQ_DEPTH - 1;
+		dev->admin_tagset.queue_depth = NVME_AQ_BLK_MQ_DEPTH - 1;
 		dev->admin_tagset.timeout = ADMIN_TIMEOUT;
 		dev->admin_tagset.numa_node = dev_to_node(dev->dev);
 		dev->admin_tagset.cmd_size = nvme_pci_cmd_size(dev, false);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index a5577e06a620..f35992b1c48b 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -41,14 +41,6 @@
 
 #define NVME_RDMA_MAX_INLINE_SEGMENTS	1
 
-/*
- * We handle AEN commands ourselves and don't even let the
- * block layer know about them.
- */
-#define NVME_RDMA_NR_AEN_COMMANDS      1
-#define NVME_RDMA_AQ_BLKMQ_DEPTH       \
-	(NVME_AQ_DEPTH - NVME_RDMA_NR_AEN_COMMANDS)
-
 struct nvme_rdma_device {
 	struct ib_device       *dev;
 	struct ib_pd	       *pd;
@@ -691,7 +683,7 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
 		set = &ctrl->admin_tag_set;
 		memset(set, 0, sizeof(*set));
 		set->ops = &nvme_rdma_admin_mq_ops;
-		set->queue_depth = NVME_RDMA_AQ_BLKMQ_DEPTH;
+		set->queue_depth = NVME_AQ_BLK_MQ_DEPTH - 1;
 		set->reserved_tags = 2; /* connect + keep-alive */
 		set->numa_node = NUMA_NO_NODE;
 		set->cmd_size = sizeof(struct nvme_rdma_request) +
@@ -1319,7 +1311,7 @@ static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 
 	memset(cmd, 0, sizeof(*cmd));
 	cmd->common.opcode = nvme_admin_async_event;
-	cmd->common.command_id = NVME_RDMA_AQ_BLKMQ_DEPTH;
+	cmd->common.command_id = NVME_AQ_BLK_MQ_DEPTH;
 	cmd->common.flags |= NVME_CMD_SGL_METABUF;
 	nvme_rdma_set_sg_null(cmd);
 
@@ -1381,7 +1373,7 @@ static int __nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc, int tag)
 	 * for them but rather special case them here.
 	 */
 	if (unlikely(nvme_rdma_queue_idx(queue) == 0 &&
-			cqe->command_id >= NVME_RDMA_AQ_BLKMQ_DEPTH))
+			cqe->command_id >= NVME_AQ_BLK_MQ_DEPTH))
 		nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status,
 				&cqe->result);
 	else
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c56354e1e4c6..fc2856943d89 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -23,14 +23,6 @@
 
 #define NVME_LOOP_MAX_SEGMENTS		256
 
-/*
- * We handle AEN commands ourselves and don't even let the
- * block layer know about them.
- */
-#define NVME_LOOP_NR_AEN_COMMANDS	1
-#define NVME_LOOP_AQ_BLKMQ_DEPTH	\
-	(NVME_AQ_DEPTH - NVME_LOOP_NR_AEN_COMMANDS)
-
 struct nvme_loop_iod {
 	struct nvme_request	nvme_req;
 	struct nvme_command	cmd;
@@ -113,7 +105,7 @@ static void nvme_loop_queue_response(struct nvmet_req *req)
 	 * for them but rather special case them here.
 	 */
 	if (unlikely(nvme_loop_queue_idx(queue) == 0 &&
-			cqe->command_id >= NVME_LOOP_AQ_BLKMQ_DEPTH)) {
+			cqe->command_id >= NVME_AQ_BLK_MQ_DEPTH)) {
 		nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status,
 				&cqe->result);
 	} else {
@@ -201,7 +193,7 @@ static void nvme_loop_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 
 	memset(&iod->cmd, 0, sizeof(iod->cmd));
 	iod->cmd.common.opcode = nvme_admin_async_event;
-	iod->cmd.common.command_id = NVME_LOOP_AQ_BLKMQ_DEPTH;
+	iod->cmd.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
 	iod->cmd.common.flags |= NVME_CMD_SGL_METABUF;
 
 	if (!nvmet_req_init(&iod->req, &queue->nvme_cq, &queue->nvme_sq,
@@ -357,7 +349,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 
 	memset(&ctrl->admin_tag_set, 0, sizeof(ctrl->admin_tag_set));
 	ctrl->admin_tag_set.ops = &nvme_loop_admin_mq_ops;
-	ctrl->admin_tag_set.queue_depth = NVME_LOOP_AQ_BLKMQ_DEPTH;
+	ctrl->admin_tag_set.queue_depth = NVME_AQ_BLK_MQ_DEPTH - 1;
 	ctrl->admin_tag_set.reserved_tags = 2; /* connect + keep-alive */
 	ctrl->admin_tag_set.numa_node = NUMA_NO_NODE;
 	ctrl->admin_tag_set.cmd_size = sizeof(struct nvme_loop_iod) +
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9310ce77d8e1..9a3878c2b2ad 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -90,6 +90,8 @@ enum {
 };
 
 #define NVME_AQ_DEPTH		32
+#define NVME_AEN_COMMANDS	1
+#define NVME_AQ_BLK_MQ_DEPTH	(NVME_AQ_DEPTH - NVME_AEN_COMMANDS)
 
 enum {
 	NVME_REG_CAP	= 0x0000,	/* Controller Capabilities */
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:23 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:23 -0600
Subject: [PATCHv2 4/5] nvme: Unexport starting async event work
In-Reply-To: <20171020221924.30659-1-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
Message-ID: <20171020221924.30659-5-keith.busch@intel.com>

Async event work is for core use only and should not be called directly
from drivers.

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 8 +-------
 drivers/nvme/host/nvme.h | 1 -
 2 files changed, 1 insertion(+), 8 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7fb55b6e26c3..74724329f430 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2670,12 +2670,6 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 }
 EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 
-void nvme_queue_async_events(struct nvme_ctrl *ctrl)
-{
-	queue_work(nvme_wq, &ctrl->async_event_work);
-}
-EXPORT_SYMBOL_GPL(nvme_queue_async_events);
-
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_stop_keep_alive(ctrl);
@@ -2692,7 +2686,7 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl)
 
 	if (ctrl->queue_count > 1) {
 		nvme_queue_scan(ctrl);
-		nvme_queue_async_events(ctrl);
+		queue_work(nvme_wq, &ctrl->async_event_work);
 		nvme_start_queues(ctrl);
 	}
 }
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 2f1e52131430..4cd4e8f4db19 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -301,7 +301,6 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 
 void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		union nvme_result *res);
-void nvme_queue_async_events(struct nvme_ctrl *ctrl);
 
 void nvme_stop_queues(struct nvme_ctrl *ctrl);
 void nvme_start_queues(struct nvme_ctrl *ctrl);
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:19:24 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:19:24 -0600
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <20171020221924.30659-1-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
Message-ID: <20171020221924.30659-6-keith.busch@intel.com>

This will give udev a chance to handle asynchronous event notification
and clear the log to unmask future events of the same type. Since the
core driver allows only one AEN request at a time, we can only have one
possible oustanding uevent to send. This implementation saves the last
AEN result from the IRQ handler, and sends the uevent change notification
when the AEN work is rescheduled.

AEN events that the kernel handles directly will not create a uevent. Such
events will stop being sent to the user as new handlers are added to
the kernel.

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 21 +++++++++++++++++++++
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 22 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 74724329f430..d05dcd172ad5 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2574,11 +2574,27 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_remove_namespaces);
 
+static void nvme_aen_uevent(struct nvme_ctrl *ctrl)
+{
+	char *envp[2] = {NULL, NULL};
+	u32 aen = ctrl->aen;
+
+	ctrl->aen = 0;
+	if (!aen)
+		return;
+
+	if (!envp[0])
+		return;
+	kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
+	kfree(envp[0]);
+}
+
 static void nvme_async_event_work(struct work_struct *work)
 {
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, async_event_work);
 
+	nvme_aen_uevent(ctrl);
 	ctrl->ops->submit_async_event(ctrl);
 }
 
@@ -2655,6 +2671,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 	if (le16_to_cpu(status) >> 1 != NVME_SC_SUCCESS)
 		return;
 
+	/*
+	 * Set result for udev only for AEN types that the kernel does not
+	 * directly handle.
+	 */
 	switch (result & 0xff07) {
 	case NVME_AER_NOTICE_NS_CHANGED:
 		dev_info(ctrl->device, "rescanning\n");
@@ -2664,6 +2684,7 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		queue_work(nvme_wq, &ctrl->fw_act_work);
 		break;
 	default:
+		ctrl->aen = result;
 		dev_warn(ctrl->device, "async event result %08x\n", result);
 	}
 	queue_work(nvme_wq, &ctrl->async_event_work);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 4cd4e8f4db19..69366293ce57 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -166,6 +166,7 @@ struct nvme_ctrl {
 	u16 kas;
 	u8 npss;
 	u8 apsta;
+	u32 aen;
 	unsigned int shutdown_timeout;
 	unsigned int kato;
 	bool subsystem;
-- 
2.13.6



From keith.busch at intel.com  Fri Oct 20 15:29:32 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:29:32 -0600
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <20171020221924.30659-6-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
 <20171020221924.30659-6-keith.busch@intel.com>
Message-ID: <20171020222932.GA30713@localhost.localdomain>

On Fri, Oct 20, 2017 at 04:19:24PM -0600, Keith Busch wrote:
> +static void nvme_aen_uevent(struct nvme_ctrl *ctrl)
> +{
> +	char *envp[2] = {NULL, NULL};
> +	u32 aen = ctrl->aen;
> +
> +	ctrl->aen = 0;
> +	if (!aen)
> +		return;
> +
> +	if (!envp[0])
> +		return;
> +	kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
> +	kfree(envp[0]);
> +}

Uh, the string format dissappeared on me. Resending...


From keith.busch at intel.com  Fri Oct 20 15:37:46 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 20 Oct 2017 16:37:46 -0600
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
Message-ID: <20171020223746.31287-1-keith.busch@intel.com>

This will give udev a chance to handle asynchronous event notification
and clear the log to unmask future events of the same type. Since the
core driver allows only one AEN request at a time, we can only have one
possible oustanding uevent to send. This implementation saves the last
AEN result from the IRQ handler, and sends the uevent change notification
when the AEN work is rescheduled.

AEN events that the kernel handles directly will not create a uevent. Such
events will stop being sent to the user as new handlers are added to
the kernel.

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 22 ++++++++++++++++++++++
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 23 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 74724329f430..e5d911dc96f4 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2574,11 +2574,28 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_remove_namespaces);
 
+static void nvme_aen_uevent(struct nvme_ctrl *ctrl)
+{
+	char *envp[2] = {NULL, NULL};
+	u32 aen = ctrl->aen;
+
+	ctrl->aen = 0;
+	if (!aen)
+		return;
+
+	envp[0] = kasprintf(GFP_KERNEL, "NVME_AEN=%#08x", aen);
+	if (!envp[0])
+		return;
+	kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
+	kfree(envp[0]);
+}
+
 static void nvme_async_event_work(struct work_struct *work)
 {
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, async_event_work);
 
+	nvme_aen_uevent(ctrl);
 	ctrl->ops->submit_async_event(ctrl);
 }
 
@@ -2655,6 +2672,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 	if (le16_to_cpu(status) >> 1 != NVME_SC_SUCCESS)
 		return;
 
+	/*
+	 * Set result for udev only for AEN types that the kernel does not
+	 * directly handle.
+	 */
 	switch (result & 0xff07) {
 	case NVME_AER_NOTICE_NS_CHANGED:
 		dev_info(ctrl->device, "rescanning\n");
@@ -2664,6 +2685,7 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		queue_work(nvme_wq, &ctrl->fw_act_work);
 		break;
 	default:
+		ctrl->aen = result;
 		dev_warn(ctrl->device, "async event result %08x\n", result);
 	}
 	queue_work(nvme_wq, &ctrl->async_event_work);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 4cd4e8f4db19..69366293ce57 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -166,6 +166,7 @@ struct nvme_ctrl {
 	u16 kas;
 	u8 npss;
 	u8 apsta;
+	u32 aen;
 	unsigned int shutdown_timeout;
 	unsigned int kato;
 	bool subsystem;
-- 
2.13.6



From james.smart at broadcom.com  Fri Oct 20 16:06:34 2017
From: james.smart at broadcom.com (James Smart)
Date: Fri, 20 Oct 2017 16:06:34 -0700
Subject: [PATCHv2 2/5] nvme/fc: remove unused "queue_size" field
In-Reply-To: <20171020221924.30659-3-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
 <20171020221924.30659-3-keith.busch@intel.com>
Message-ID: <bb21869c-0e51-af12-1070-c7f0c6de30d3@broadcom.com>

On 10/20/2017 3:19 PM, Keith Busch wrote:
> This was being saved in a structure, but never used anywhere. The queue
> size is obtained through other means, so there's no reason to duplicate
> this without a user for it.
>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> Signed-off-by: Keith Busch <keith.busch at intel.com>
> ---
>   drivers/nvme/host/fc.c | 9 +++------
>

Looks good. This stuff was left over when queuesize moved into the nvme 
ctrl.

-- james

Reviewed-by: James Smart <james.smart at broadcom.com>



From jsmart2021 at gmail.com  Fri Oct 20 16:17:07 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:17:07 -0700
Subject: [PATCH v2 2/4] nvme: add helper to compare options to controller
In-Reply-To: <20171020231709.3183-1-jsmart2021@gmail.com>
References: <20171020231709.3183-1-jsmart2021@gmail.com>
Message-ID: <20171020231709.3183-3-jsmart2021@gmail.com>

Adds a helper function that compares the host and subsytem
specified in a connect options list vs a controller.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fabrics.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index a7590cc59ba2..42232e731f19 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -133,6 +133,18 @@ struct nvmf_transport_ops {
 					struct nvmf_ctrl_options *opts);
 };
 
+static inline bool
+nvmf_ctlr_matches_baseopts(struct nvme_ctrl *ctrl,
+			struct nvmf_ctrl_options *opts)
+{
+	if (strcmp(opts->subsysnqn, ctrl->opts->subsysnqn) ||
+	    strcmp(opts->host->nqn, ctrl->opts->host->nqn) ||
+	    memcmp(&opts->host->id, &ctrl->opts->host->id, sizeof(uuid_t)))
+		return false;
+
+	return true;
+}
+
 int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val);
 int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val);
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 16:17:05 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:17:05 -0700
Subject: [PATCH v2 0/4] Add duplicate_connect option for ctlr connect requests
Message-ID: <20171020231709.3183-1-jsmart2021@gmail.com>

This patch adds a duplicate_connect option option to those supported by
fabric connection requests. When set to true (1), duplicate connections
are allowed. When set to false (0), duplicate connections are rejected.

The option is set in the fabrics layer and acted upon by each transport.

This patch set came about as of this prior patch review:
http://lists.infradead.org/pipermail/linux-nvme/2017-September/012783.html

---
v2:
 split compare for hostnqn, hostid, subnqn to a helper function
 duplicate_connection converted to a boolean option (presence means true)

James Smart (4):
  nvme: add duplicate_connect option
  nvme: add helper to compare options to controller
  nvme_fc: add support for duplicate_connect option
  RFC: nvme-rdma: add support for duplicate_connect option

 drivers/nvme/host/fabrics.c |  7 +++-
 drivers/nvme/host/fabrics.h | 14 ++++++++
 drivers/nvme/host/fc.c      | 33 ++++++++++++++++++
 drivers/nvme/host/rdma.c    | 82 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 135 insertions(+), 1 deletion(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 16:17:08 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:17:08 -0700
Subject: [PATCH v2 3/4] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171020231709.3183-1-jsmart2021@gmail.com>
References: <20171020231709.3183-1-jsmart2021@gmail.com>
Message-ID: <20171020231709.3183-4-jsmart2021@gmail.com>

Adds support for the duplicate_connect option. When set to true,
checks whether there's an existing controller via the same host port
and target port for the same host (hostnqn, hostid) to the same
subsystem. Fails the connection request if an existing controller.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 0d0e898912ad..90dd01811dfb 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2785,6 +2785,33 @@ static const struct blk_mq_ops nvme_fc_admin_mq_ops = {
 };
 
 
+/*
+ * Fails a controller request if it matches an existing controller
+ * (association) with the same tuple:
+ * <Host NQN, Host ID, local FC port, remote FC port, SUBSYS NQN>
+ *
+ * The ports don't need to be compared as they are intrinsically
+ * already matched by the port pointers supplied.
+ */
+static bool
+nvme_fc_existing_controller(struct nvme_fc_rport *rport,
+		struct nvmf_ctrl_options *opts)
+{
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+	bool found = false;
+
+	spin_lock_irqsave(&rport->lock, flags);
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
+		found = nvmf_ctlr_matches_baseopts(&ctrl->ctrl, opts);
+		if (found)
+			break;
+	}
+	spin_unlock_irqrestore(&rport->lock, flags);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	struct nvme_fc_lport *lport, struct nvme_fc_rport *rport)
@@ -2799,6 +2826,12 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 		goto out_fail;
 	}
 
+	if (!opts->duplicate_connect &&
+	    nvme_fc_existing_controller(rport, opts)) {
+		ret = -EALREADY;
+		goto out_fail;
+	}
+
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl) {
 		ret = -ENOMEM;
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 16:17:06 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:17:06 -0700
Subject: [PATCH v2 1/4] nvme: add duplicate_connect option
In-Reply-To: <20171020231709.3183-1-jsmart2021@gmail.com>
References: <20171020231709.3183-1-jsmart2021@gmail.com>
Message-ID: <20171020231709.3183-2-jsmart2021@gmail.com>

Add the "duplicate_connect" boolean option (presence means true).
Default is false.

When false, the transport should validate whether a new controller request
is targeted for the same host transport addressing and target transport
addressing as an existing controller. If so, the new controller request
should be rejected.

When true, the callee is explicitly requesting a duplicate controller
connection to be made and the new request should be attempted.

Signed-off-by: James Smart <james.smart at broadcom.com>

--
v2:
 duplicate_connect option converted to a boolean
---
 drivers/nvme/host/fabrics.c | 7 ++++++-
 drivers/nvme/host/fabrics.h | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 8bca36a46924..4a83137b0268 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -548,6 +548,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_HOSTNQN,		"hostnqn=%s"		},
 	{ NVMF_OPT_HOST_TRADDR,		"host_traddr=%s"	},
 	{ NVMF_OPT_HOST_ID,		"hostid=%s"		},
+	{ NVMF_OPT_DUP_CONNECT,		"duplicate_connect"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -566,6 +567,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	opts->nr_io_queues = num_online_cpus();
 	opts->reconnect_delay = NVMF_DEF_RECONNECT_DELAY;
 	opts->kato = NVME_DEFAULT_KATO;
+	opts->duplicate_connect = false;
 
 	options = o = kstrdup(buf, GFP_KERNEL);
 	if (!options)
@@ -742,6 +744,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 				goto out;
 			}
 			break;
+		case NVMF_OPT_DUP_CONNECT:
+			opts->duplicate_connect = true;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -823,7 +828,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 #define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
 #define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID)
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index bf33663218cd..a7590cc59ba2 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -57,6 +57,7 @@ enum {
 	NVMF_OPT_HOST_TRADDR	= 1 << 10,
 	NVMF_OPT_CTRL_LOSS_TMO	= 1 << 11,
 	NVMF_OPT_HOST_ID	= 1 << 12,
+	NVMF_OPT_DUP_CONNECT	= 1 << 13,
 };
 
 /**
@@ -96,6 +97,7 @@ struct nvmf_ctrl_options {
 	unsigned int		nr_io_queues;
 	unsigned int		reconnect_delay;
 	bool			discovery_nqn;
+	bool			duplicate_connect;
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 16:17:09 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:17:09 -0700
Subject: [PATCH v2 4/4] nvme-rdma: add support for duplicate_connect option
In-Reply-To: <20171020231709.3183-1-jsmart2021@gmail.com>
References: <20171020231709.3183-1-jsmart2021@gmail.com>
Message-ID: <20171020231709.3183-5-jsmart2021@gmail.com>

Adds support for the duplicate_connect option. When set to true,
checks whether there's an existing controller via the same target
address (traddr), target port (trsvcid), and if specified, host
address (host_traddr). Fails the connection request if there is
an existing controller.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/rdma.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index a5577e06a620..523881444fee 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1848,6 +1848,83 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.reinit_request		= nvme_rdma_reinit_request,
 };
 
+static inline bool
+__nvme_rdma_options_match(struct nvme_rdma_ctrl *ctrl,
+	struct nvmf_ctrl_options *opts)
+{
+	char *stdport = __stringify(NVME_RDMA_IP_PORT);
+
+
+	if (!nvmf_ctlr_matches_baseopts(&ctrl->ctrl, opts) ||
+	    strcmp(opts->traddr, ctrl->ctrl.opts->traddr))
+		return false;
+
+	if (opts->mask & NVMF_OPT_TRSVCID &&
+	    ctrl->ctrl.opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(opts->trsvcid, ctrl->ctrl.opts->trsvcid))
+			return false;
+	} else if (opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(opts->trsvcid, stdport))
+			return false;
+	} else if (ctrl->ctrl.opts->mask & NVMF_OPT_TRSVCID) {
+		if (strcmp(stdport, ctrl->ctrl.opts->trsvcid))
+			return false;
+	}
+	/* else, it's a match as both have stdport. Fall to next checks */
+
+	/*
+	 * checking the local address is rough. In most cases, one
+	 * is not specified and the host port is selected by the stack.
+	 *
+	 * Assume no match if:
+	 *  local address is specified and address is not the same
+	 *  local address is not specified but remote is, or vice versa
+	 *    (admin using specific host_traddr when it matters).
+	 */
+	if (opts->mask & NVMF_OPT_HOST_TRADDR &&
+	    ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR) {
+		if (strcmp(opts->host_traddr, ctrl->ctrl.opts->host_traddr))
+			return false;
+	} else if (opts->mask & NVMF_OPT_HOST_TRADDR ||
+		   ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR)
+		return false;
+	/*
+	 * if neither controller had an host port specified, assume it's
+	 * a match as everything else matched.
+	 */
+
+	return true;
+}
+
+/*
+ * Fails a connection request if it matches an existing controller
+ * (association) with the same tuple:
+ * <Host NQN, Host ID, local address, remote address, remote port, SUBSYS NQN>
+ *
+ * if local address is not specified in the request, it will match an
+ * existing controller with all the other parameters the same and no
+ * local port address specified as well.
+ *
+ * The ports don't need to be compared as they are intrinsically
+ * already matched by the port pointers supplied.
+ */
+static bool
+nvme_rdma_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_rdma_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_rdma_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_rdma_ctrl_list, list) {
+		found = __nvme_rdma_options_match(ctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_rdma_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
@@ -1884,6 +1961,11 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 		}
 	}
 
+	if (!opts->duplicate_connect && nvme_rdma_existing_controller(opts)) {
+		ret = -EALREADY;
+		goto out_free_ctrl;
+	}
+
 	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops,
 				0 /* no quirks, we're perfect! */);
 	if (ret)
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 20 16:18:15 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 20 Oct 2017 16:18:15 -0700
Subject: [PATCH v2] nvme-cli: add duplicate_connect parameter support
Message-ID: <20171020231815.3288-1-jsmart2021@gmail.com>

Add support for the duplicate_connect parameter for the fabrics
device.

Rather than add yet another 6-line code block, created a small helper
routine invoked for each argument option.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2:
  converted to a boolean arg (true if present).
  helper expanded for different arg types
---
 fabrics.c | 140 ++++++++++++++++++++++++--------------------------------------
 1 file changed, 54 insertions(+), 86 deletions(-)

diff --git a/fabrics.c b/fabrics.c
index 5dec9f4..4bdaaf2 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -61,6 +61,7 @@ static struct config {
 	char *ctrl_loss_tmo;
 	char *raw;
 	char *device;
+	int  duplicate_connect;
 } cfg = { NULL };
 
 #define BUF_SIZE		4096
@@ -484,6 +485,38 @@ out:
 	return ret;
 }
 
+static int
+add_bool_argument(char **argstr, int *max_len, char *arg_str, bool arg)
+{
+	int len;
+
+	if (arg) {
+		len = snprintf(*argstr, *max_len, ",%s", arg_str);
+		if (len < 0)
+			return -EINVAL;
+		*argstr += len;
+		*max_len -= len;
+	}
+
+	return 0;
+}
+
+static int
+add_argument(char **argstr, int *max_len, char *arg_str, char *arg)
+{
+	int len;
+
+	if (arg) {
+		len = snprintf(*argstr, *max_len, ",%s=%s", arg_str, arg);
+		if (len < 0)
+			return -EINVAL;
+		*argstr += len;
+		*max_len -= len;
+	}
+
+	return 0;
+}
+
 static int build_options(char *argstr, int max_len)
 {
 	int len;
@@ -500,99 +533,33 @@ static int build_options(char *argstr, int max_len)
 		}
 	}
 
+	/* always specify nqn as first arg - this will init the string */
 	len = snprintf(argstr, max_len, "nqn=%s", cfg.nqn);
 	if (len < 0)
 		return -EINVAL;
 	argstr += len;
 	max_len -= len;
 
-	len = snprintf(argstr, max_len, ",transport=%s", cfg.transport);
-	if (len < 0)
+	if (add_argument(&argstr, &max_len, "transport", cfg.transport) ||
+	    add_argument(&argstr, &max_len, "traddr", cfg.traddr) ||
+	    add_argument(&argstr, &max_len, "host_traddr", cfg.host_traddr) ||
+	    add_argument(&argstr, &max_len, "trsvcid", cfg.trsvcid) ||
+	    ((cfg.hostnqn || nvmf_hostnqn_file()) &&
+		    add_argument(&argstr, &max_len, "hostnqn", cfg.hostnqn)) ||
+	    ((cfg.hostid || nvmf_hostid_file()) &&
+		    add_argument(&argstr, &max_len, "hostid", cfg.hostid)) ||
+	    add_argument(&argstr, &max_len, "nr_io_queues",
+				cfg.nr_io_queues) ||
+	    add_argument(&argstr, &max_len, "queue_size", cfg.queue_size) ||
+	    add_argument(&argstr, &max_len, "keep_alive_tmo",
+				cfg.keep_alive_tmo) ||
+	    add_argument(&argstr, &max_len, "reconnect_delay",
+				cfg.reconnect_delay) ||
+	    add_argument(&argstr, &max_len, "ctrl_loss_tmo",
+				cfg.ctrl_loss_tmo) ||
+	    add_bool_argument(&argstr, &max_len, "duplicate_connect",
+				cfg.duplicate_connect))
 		return -EINVAL;
-	argstr += len;
-	max_len -= len;
-
-	if (cfg.traddr) {
-		len = snprintf(argstr, max_len, ",traddr=%s", cfg.traddr);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.host_traddr) {
-		len = snprintf(argstr, max_len, ",host_traddr=%s", cfg.host_traddr);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.trsvcid) {
-		len = snprintf(argstr, max_len, ",trsvcid=%s", cfg.trsvcid);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.hostnqn || nvmf_hostnqn_file()) {
-		len = snprintf(argstr, max_len, ",hostnqn=%s", cfg.hostnqn);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.hostid || nvmf_hostid_file()) {
-		len = snprintf(argstr, max_len, ",hostid=%s", cfg.hostid);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.nr_io_queues) {
-		len = snprintf(argstr, max_len, ",nr_io_queues=%s",
-				cfg.nr_io_queues);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.queue_size) {
-		len = snprintf(argstr, max_len, ",queue_size=%s",
-				cfg.queue_size);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.keep_alive_tmo) {
-		len = snprintf(argstr, max_len, ",keep_alive_tmo=%s", cfg.keep_alive_tmo);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.reconnect_delay) {
-		len = snprintf(argstr, max_len, ",reconnect_delay=%s", cfg.reconnect_delay);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
-
-	if (cfg.ctrl_loss_tmo) {
-		len = snprintf(argstr, max_len, ",ctrl_loss_tmo=%s", cfg.ctrl_loss_tmo);
-		if (len < 0)
-			return -EINVAL;
-		argstr += len;
-		max_len -= len;
-	}
 
 	return 0;
 }
@@ -888,6 +855,7 @@ int connect(const char *desc, int argc, char **argv)
 		{"keep-alive-tmo",  'k', "LIST", CFG_STRING, &cfg.keep_alive_tmo,  required_argument, "keep alive timeout period in seconds" },
 		{"reconnect-delay", 'c', "LIST", CFG_STRING, &cfg.reconnect_delay, required_argument, "reconnect timeout period in seconds" },
 		{"ctrl-loss-tmo",   'l', "LIST", CFG_STRING, &cfg.ctrl_loss_tmo,   required_argument, "controller loss timeout period in seconds" },
+		{"duplicate_connect", 'D', "", CFG_NONE, &cfg.duplicate_connect, no_argument, "allow duplicate connections between same transport host and subsystem port" },
 		{NULL},
 	};
 
-- 
2.13.1



From minwoo.im.dev at gmail.com  Fri Oct 20 21:23:16 2017
From: minwoo.im.dev at gmail.com (Minwoo Im)
Date: Sat, 21 Oct 2017 13:23:16 +0900
Subject: [PATCH] nvme-cli: make address print format more human-readable
Message-ID: <1508559796-7605-1-git-send-email-minwoo.im.dev@gmail.com>

Memory address would be great in hexadecimal format to be read easily.

Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>
---
 nvme-print.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/nvme-print.c b/nvme-print.c
index 9636467..df4ed59 100644
--- a/nvme-print.c
+++ b/nvme-print.c
@@ -1255,8 +1255,8 @@ static void show_auto_pst(struct nvme_auto_pst *apst)
 static void show_host_mem_buffer(struct nvme_host_mem_buffer *hmb)
 {
 	printf("\tHost Memory Descriptor List Entry Count (HMDLEC): %u\n", hmb->hmdlec);
-	printf("\tHost Memory Descriptor List Address     (HMDLAU): %u\n", hmb->hmdlau);
-	printf("\tHost Memory Descriptor List Address     (HMDLAL): %u\n", hmb->hmdlal);
+	printf("\tHost Memory Descriptor List Address     (HMDLAU): 0x%x\n", hmb->hmdlau);
+	printf("\tHost Memory Descriptor List Address     (HMDLAL): 0x%x\n", hmb->hmdlal);
 	printf("\tHost Memory Buffer Size                  (HSIZE): %u\n", hmb->hsize);
 }
 
-- 
2.7.4



From hch at lst.de  Sat Oct 21 01:00:45 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 21 Oct 2017 10:00:45 +0200
Subject: [PATCH] nvme: Remove unused headers
In-Reply-To: <20171020221803.30484-1-keith.busch@intel.com>
References: <20171020221803.30484-1-keith.busch@intel.com>
Message-ID: <20171021080045.GA20906@lst.de>

Looks fine,

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at lst.de  Sat Oct 21 01:06:19 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 21 Oct 2017 10:06:19 +0200
Subject: [PATCHv2 1/5] nvme: Centralize AEN defines
In-Reply-To: <20171020221924.30659-2-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
 <20171020221924.30659-2-keith.busch@intel.com>
Message-ID: <20171021080619.GB20906@lst.de>

> +	ctrl->event_limit = NVME_AEN_COMMANDS;

Can we keep the NR_? E.g. NVME_NR_AEN_COMMANDS?

> +	struct nvme_fc_fcp_op	aen_ops[NVME_AEN_COMMANDS];
>  
>  	struct nvme_ctrl	ctrl;
>  };
> @@ -1322,7 +1313,7 @@ nvme_fc_abort_aen_ops(struct nvme_fc_ctrl *ctrl)
>  	unsigned long flags;
>  	int i, ret;
>  
> -	for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
> +	for (i = 0; i < NVME_AEN_COMMANDS; i++, aen_op++) {
>  		if (atomic_read(&aen_op->state) != FCPOP_STATE_ACTIVE)
>  			continue;

Kill the array and loop?

> -	ctrl->admin_tag_set.queue_depth = NVME_FC_AQ_BLKMQ_DEPTH;
> +	ctrl->admin_tag_set.queue_depth = NVME_AQ_BLK_MQ_DEPTH - 1;

Shouldn't the -1 be factored into NVME_AQ_BLK_MQ_DEPTH?  Either way
I think we need some constant for it, and move the explanation for it
from the PCI driver to the core somehow.


From hch at lst.de  Sat Oct 21 01:06:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 21 Oct 2017 10:06:55 +0200
Subject: [PATCHv2 3/5] nvme: Single AEN request
In-Reply-To: <20171020221924.30659-4-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
 <20171020221924.30659-4-keith.busch@intel.com>
Message-ID: <20171021080655.GC20906@lst.de>

Looks fine,

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at lst.de  Sat Oct 21 01:07:43 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 21 Oct 2017 10:07:43 +0200
Subject: [PATCHv2 4/5] nvme: Unexport starting async event work
In-Reply-To: <20171020221924.30659-5-keith.busch@intel.com>
References: <20171020221924.30659-1-keith.busch@intel.com>
 <20171020221924.30659-5-keith.busch@intel.com>
Message-ID: <20171021080743.GD20906@lst.de>

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at lst.de  Sat Oct 21 01:09:06 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 21 Oct 2017 10:09:06 +0200
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <20171020223746.31287-1-keith.busch@intel.com>
References: <20171020223746.31287-1-keith.busch@intel.com>
Message-ID: <20171021080906.GE20906@lst.de>

On Fri, Oct 20, 2017 at 04:37:46PM -0600, Keith Busch wrote:
> This will give udev a chance to handle asynchronous event notification
> and clear the log to unmask future events of the same type. Since the
> core driver allows only one AEN request at a time, we can only have one
> possible oustanding uevent to send.

I don't want to encode that in the userspace ABI.  With all kinds of
new AEN uses coming up we might as well want to go for multiple AERs
in the future.

> This implementation saves the last
> AEN result from the IRQ handler, and sends the uevent change notification
> when the AEN work is rescheduled.
> 
> AEN events that the kernel handles directly will not create a uevent. Such
> events will stop being sent to the user as new handlers are added to
> the kernel.

I think we need an explicit whitelist of the ones we want to expose
to userspace.  Otherwise we'll get all kinds of complaints later
that we'd take something away and break the ABI.


From nitzanc at mellanox.com  Sun Oct 22 02:37:04 2017
From: nitzanc at mellanox.com (Nitzan Carmi)
Date: Sun, 22 Oct 2017 09:37:04 +0000
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
Message-ID: <20171022093704.25203-1-nitzanc@mellanox.com>

Signed-off-by: Nitzan Carmi <nitzanc at mellanox.com>
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/host/rdma.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..f2f9a47 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1582,6 +1582,10 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
 {
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
 
+	dev_warn(req->queue->ctrl->ctrl.device,
+		 "I/O %d QID %d timeout, recover controller\n",
+		 rq->tag, nvme_rdma_queue_idx(req->queue));
+
 	/* queue error recovery */
 	nvme_rdma_error_recovery(req->queue->ctrl);
 
-- 
2.9.4



From sagi at grimberg.me  Sun Oct 22 03:05:19 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 13:05:19 +0300
Subject: [PATCH] nvme_fc: avoid workqueue flush stalls
In-Reply-To: <327c6f13-0bd3-5e95-a9e5-d104c561680d@broadcom.com>
References: <20170928165631.16097-1-jsmart2021@gmail.com>
 <01c52881-b172-0d86-5db2-596acf6394db@grimberg.me>
 <327c6f13-0bd3-5e95-a9e5-d104c561680d@broadcom.com>
Message-ID: <56c72dfe-132f-f219-3d50-c7775a84a209@grimberg.me>


>>> There's no need to wait for the full nvme_wq, which is now shared,
>>> to flush. flush only the delete_work item.
>>
>> The patch looks correct, but can you explain what stalled that this
>> patch fixes?
> 
> I don't have an explicit failure case. While reviewing the code for the 
> other stall issue with fatal_err_work, I realized that this flush is now 
> waiting on a work queue shared by everything nvmf.? It used to be solely 
> a nvme-fc transport workqueue. There's no need for waiting on anything 
> more than the delete work item. And having it raise in scope of the nvmf 
> thread perhaps makes it wait for a lot of other unnecessary things - 
> slowing it down.
> 
> If you're ok with it - can you Ack/Review it ?

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Sun Oct 22 03:05:46 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 13:05:46 +0300
Subject: [PATCH] nvme: Remove unused headers
In-Reply-To: <20171020221803.30484-1-keith.busch@intel.com>
References: <20171020221803.30484-1-keith.busch@intel.com>
Message-ID: <37434503-6f60-d9be-2916-7042d04771b9@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Sun Oct 22 04:11:47 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 14:11:47 +0300
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl setup/teardown
 symmetric
In-Reply-To: <0a081ffc-5a3d-6b52-7257-18e39034ce93@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171019152910.GA24808@lst.de>
 <0a081ffc-5a3d-6b52-7257-18e39034ce93@grimberg.me>
Message-ID: <d1e113b8-1f76-1084-b075-a32d484287d8@grimberg.me>


>>> From: Roy Shterman <roys at lightbitslabs.com>
>>>
>>> Its strange to have some of the cleanup for nvme_init_ctrl in
>>> nvme_free_ctrl. This cause us to have some magic ctrl refcount
>>> handling in controller initialization error paths.
>>>
>>> Make the init/uninit symmetric so we can have a sane and
>>> maintainable error sequences.
>>
>> This totally conflicts with my refcount / device rework in the
>> mpath patchset..
> 
> I know...
> 
> Do you want me to rebase on top of your next round of patches?

ping?


From sagi at grimberg.me  Sun Oct 22 10:16:37 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 20:16:37 +0300
Subject: v4.14-rc5 NVMeOF regression?
In-Reply-To: <1508343856.2540.8.camel@wdc.com>
References: <1508192588.2493.60.camel@wdc.com>
 <82de6177-27f5-463e-3446-0305fd9f6790@grimberg.me>
 <5fb38923-36f7-c069-5f1d-96f4a9c98248@wdc.com>
 <07f75a37-cd25-81c1-3264-2be80bb9a763@grimberg.me>
 <1508343856.2540.8.camel@wdc.com>
Message-ID: <9b7da98b-c3f0-c076-574e-0a70f5566913@grimberg.me>


>> If you ran into a real deadlock, did you have any other output from
>> hung_task watchdog? I do not yet understand the root cause from
>> lockdep info provided.
>>
>> Also, do you know at which test-case this happened?
> 
> Hello Sagi,
> 
> Running test case 1 should be sufficient to trigger the deadlock. SysRq-w
> produced the following output:
> 
> sysrq: SysRq : Show Blocked State
>    task                        PC stack   pid father
> kworker/u66:2   D    0   440      2 0x80000000
> Workqueue: nvme-wq nvme_rdma_del_ctrl_work [nvme_rdma]
> Call Trace:
>   __schedule+0x3e9/0xb00
>   schedule+0x40/0x90
>   schedule_timeout+0x221/0x580
>   io_schedule_timeout+0x1e/0x50
>   wait_for_completion_io_timeout+0x118/0x180
>   blk_execute_rq+0x86/0xc0
>   __nvme_submit_sync_cmd+0x89/0xf0
>   nvmf_reg_write32+0x4b/0x90 [nvme_fabrics]
>   nvme_shutdown_ctrl+0x41/0xe0
>   nvme_rdma_shutdown_ctrl+0xca/0xd0 [nvme_rdma]
>   nvme_rdma_remove_ctrl+0x2b/0x40 [nvme_rdma]
>   nvme_rdma_del_ctrl_work+0x25/0x30 [nvme_rdma]
>   process_one_work+0x1fd/0x630
>   worker_thread+0x1db/0x3b0
>   kthread+0x11e/0x150
>   ret_from_fork+0x27/0x40
> 01              D    0  2868   2862 0x00000000
> Call Trace:
>   __schedule+0x3e9/0xb00
>   schedule+0x40/0x90
>   schedule_timeout+0x260/0x580
>   wait_for_completion+0x108/0x170
>   flush_work+0x1e0/0x270
>   nvme_rdma_del_ctrl+0x5a/0x80 [nvme_rdma]
>   nvme_sysfs_delete+0x2a/0x40
>   dev_attr_store+0x18/0x30
>   sysfs_kf_write+0x45/0x60
>   kernfs_fop_write+0x124/0x1c0
>   __vfs_write+0x28/0x150
>   vfs_write+0xc7/0x1b0
>   SyS_write+0x49/0xa0
>   entry_SYSCALL_64_fastpath+0x18/0xad

Hi Bart,

So I've looked into this, and I want to share my findings.
I'm able to reproduce this hang when trying to disconnect from
a controller which is already in reconnecting state.

The problem as I see it, is that we are returning BLK_STS_RESOURCE
from nvme_rdma_queue_rq() before we start the request timer (fail
before blk_mq_start_request) so the request timeout never
expires (and given that we are in deleting sequence, the command is
never expected to complete).

But for some reason, I don't see the request get re-issued again,
should the driver take care of this by calling
blk_mq_delay_run_hw_queue()?

Thinking more on this, if we are disconnecting from a controller,
and we are unable to issue admin/io commands (queue state is not
LIVE) we probably should not fail with BLK_STS_RESOURCE but rather
BLK_STS_IOERR.

This change makes the issue go away:
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 5b5458012c2c..be77cd098182 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1393,6 +1393,12 @@ nvme_rdma_queue_is_ready(struct nvme_rdma_queue 
*queue, struct request *rq)
                     cmd->common.opcode != nvme_fabrics_command ||
                     cmd->fabrics.fctype != nvme_fabrics_type_connect) {
                         /*
+                        * deleting state means that the ctrl will never 
accept
+                        * commands again, fail it permanently.
+                        */
+                       if (queue->ctrl->ctrl.state == NVME_CTRL_DELETING)
+                               return BLK_STS_IOERR;
+                       /*
                          * reconnecting state means transport 
disruption, which
                          * can take a long time and even might fail 
permanently,
                          * so we can't let incoming I/O be requeued 
forever.
--

Does anyone have a better idea?


From sagi at grimberg.me  Sun Oct 22 11:42:51 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 21:42:51 +0300
Subject: [PATCH] nvme-rdma: fix possible hang when issuing commands during
 ctrl removal
Message-ID: <1508697771-3552-1-git-send-email-sagi@grimberg.me>

nvme_rdma_queue_is_ready() fails requests in case a queue is not
LIVE. If the controller is in RECONNECTING state, we might be in
this state for a long time (until we successfully reconnect) and
we are better off with failing the request fast. Otherwise, we
fail with BLK_STS_RESOURCE to have the block layer try again
soon.

In case we are removing the controller when the admin queue
is not LIVE, we will terminate the request with BLK_STS_RESOURCE
but it happens before we call blk_mq_start_request() so the
request timeout never expires, and the queue will never get
back to LIVE (because we are removing the controller). This
causes the removal operation to block infinitly [1].

Thus, if we are removing (state DELETING), and the queue is
not LIVE, we need to fail the request permanently as there is
no chance for it to ever complete successfully.

[1]
--
sysrq: SysRq : Show Blocked State
  task                        PC stack   pid father
kworker/u66:2   D    0   440      2 0x80000000
Workqueue: nvme-wq nvme_rdma_del_ctrl_work [nvme_rdma]
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x221/0x580
 io_schedule_timeout+0x1e/0x50
 wait_for_completion_io_timeout+0x118/0x180
 blk_execute_rq+0x86/0xc0
 __nvme_submit_sync_cmd+0x89/0xf0
 nvmf_reg_write32+0x4b/0x90 [nvme_fabrics]
 nvme_shutdown_ctrl+0x41/0xe0
 nvme_rdma_shutdown_ctrl+0xca/0xd0 [nvme_rdma]
 nvme_rdma_remove_ctrl+0x2b/0x40 [nvme_rdma]
 nvme_rdma_del_ctrl_work+0x25/0x30 [nvme_rdma]
 process_one_work+0x1fd/0x630
 worker_thread+0x1db/0x3b0
 kthread+0x11e/0x150
 ret_from_fork+0x27/0x40
01              D    0  2868   2862 0x00000000
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x260/0x580
 wait_for_completion+0x108/0x170
 flush_work+0x1e0/0x270
 nvme_rdma_del_ctrl+0x5a/0x80 [nvme_rdma]
 nvme_sysfs_delete+0x2a/0x40
 dev_attr_store+0x18/0x30
 sysfs_kf_write+0x45/0x60
 kernfs_fop_write+0x124/0x1c0
 __vfs_write+0x28/0x150
 vfs_write+0xc7/0x1b0
 SyS_write+0x49/0xa0
 entry_SYSCALL_64_fastpath+0x18/0xad
--

Reported-by: Bart Van Assche <bart.vanassche at wdc.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 5b5458012c2c..57f1c9f49a54 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1393,6 +1393,14 @@ nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
 		    cmd->common.opcode != nvme_fabrics_command ||
 		    cmd->fabrics.fctype != nvme_fabrics_type_connect) {
 			/*
+			 * deleting state means that the ctrl will never accept
+			 * commands again, fail it permanently.
+			 */
+			if (queue->ctrl->ctrl.state == NVME_CTRL_DELETING) {
+				nvme_req(rq)->status = NVME_SC_ABORT_REQ;
+				return BLK_STS_IOERR;
+			}
+			/*
 			 * reconnecting state means transport disruption, which
 			 * can take a long time and even might fail permanently,
 			 * so we can't let incoming I/O be requeued forever.
-- 
2.7.4



From sagi at grimberg.me  Sun Oct 22 11:54:39 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 22 Oct 2017 21:54:39 +0300
Subject: [PATCH] nvme-rdma: fix possible hang when issuing commands during
 ctrl removal
In-Reply-To: <1508697771-3552-1-git-send-email-sagi@grimberg.me>
References: <1508697771-3552-1-git-send-email-sagi@grimberg.me>
Message-ID: <541a7b75-f382-70d3-442c-48a00f3bb9d1@grimberg.me>

Replaced Bart's old email address (now updated in .gitaliases)

On 22/10/17 21:42, Sagi Grimberg wrote:
> nvme_rdma_queue_is_ready() fails requests in case a queue is not
> LIVE. If the controller is in RECONNECTING state, we might be in
> this state for a long time (until we successfully reconnect) and
> we are better off with failing the request fast. Otherwise, we
> fail with BLK_STS_RESOURCE to have the block layer try again
> soon.
> 
> In case we are removing the controller when the admin queue
> is not LIVE, we will terminate the request with BLK_STS_RESOURCE
> but it happens before we call blk_mq_start_request() so the
> request timeout never expires, and the queue will never get
> back to LIVE (because we are removing the controller). This
> causes the removal operation to block infinitly [1].
> 
> Thus, if we are removing (state DELETING), and the queue is
> not LIVE, we need to fail the request permanently as there is
> no chance for it to ever complete successfully.
> 
> [1]
> --
> sysrq: SysRq : Show Blocked State
>    task                        PC stack   pid father
> kworker/u66:2   D    0   440      2 0x80000000
> Workqueue: nvme-wq nvme_rdma_del_ctrl_work [nvme_rdma]
> Call Trace:
>   __schedule+0x3e9/0xb00
>   schedule+0x40/0x90
>   schedule_timeout+0x221/0x580
>   io_schedule_timeout+0x1e/0x50
>   wait_for_completion_io_timeout+0x118/0x180
>   blk_execute_rq+0x86/0xc0
>   __nvme_submit_sync_cmd+0x89/0xf0
>   nvmf_reg_write32+0x4b/0x90 [nvme_fabrics]
>   nvme_shutdown_ctrl+0x41/0xe0
>   nvme_rdma_shutdown_ctrl+0xca/0xd0 [nvme_rdma]
>   nvme_rdma_remove_ctrl+0x2b/0x40 [nvme_rdma]
>   nvme_rdma_del_ctrl_work+0x25/0x30 [nvme_rdma]
>   process_one_work+0x1fd/0x630
>   worker_thread+0x1db/0x3b0
>   kthread+0x11e/0x150
>   ret_from_fork+0x27/0x40
> 01              D    0  2868   2862 0x00000000
> Call Trace:
>   __schedule+0x3e9/0xb00
>   schedule+0x40/0x90
>   schedule_timeout+0x260/0x580
>   wait_for_completion+0x108/0x170
>   flush_work+0x1e0/0x270
>   nvme_rdma_del_ctrl+0x5a/0x80 [nvme_rdma]
>   nvme_sysfs_delete+0x2a/0x40
>   dev_attr_store+0x18/0x30
>   sysfs_kf_write+0x45/0x60
>   kernfs_fop_write+0x124/0x1c0
>   __vfs_write+0x28/0x150
>   vfs_write+0xc7/0x1b0
>   SyS_write+0x49/0xa0
>   entry_SYSCALL_64_fastpath+0x18/0xad
> --
> 
> Reported-by: Bart Van Assche <bart.vanassche at wdc.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   drivers/nvme/host/rdma.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 5b5458012c2c..57f1c9f49a54 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -1393,6 +1393,14 @@ nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
>   		    cmd->common.opcode != nvme_fabrics_command ||
>   		    cmd->fabrics.fctype != nvme_fabrics_type_connect) {
>   			/*
> +			 * deleting state means that the ctrl will never accept
> +			 * commands again, fail it permanently.
> +			 */
> +			if (queue->ctrl->ctrl.state == NVME_CTRL_DELETING) {
> +				nvme_req(rq)->status = NVME_SC_ABORT_REQ;
> +				return BLK_STS_IOERR;
> +			}
> +			/*
>   			 * reconnecting state means transport disruption, which
>   			 * can take a long time and even might fail permanently,
>   			 * so we can't let incoming I/O be requeued forever.
> 


From guanjunxiong at huawei.com  Sun Oct 22 19:08:52 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Mon, 23 Oct 2017 10:08:52 +0800
Subject: nvme multipath support V4
In-Reply-To: <20171018165258.23212-1-hch@lst.de>
References: <20171018165258.23212-1-hch@lst.de>
Message-ID: <fbccc0db-77cd-fabc-0225-4b515729f214@huawei.com>

Hi Christoph,


On 2017/10/19 0:52, Christoph Hellwig wrote:
> Hi all,
> 
> this series adds support for multipathing, that is accessing nvme
> namespaces through multiple controllers to the nvme core driver.
> 
> It is a very thin and efficient implementation that relies on
> close cooperation with other bits of the nvme driver, and few small
> and simple block helpers.
> 
> Compared to dm-multipath the important differences are how management
> of the paths is done, and how the I/O path works.
> 
> Management of the paths is fully integrated into the nvme driver,
> for each newly found nvme controller we check if there are other
> controllers that refer to the same subsystem, and if so we link them
> up in the nvme driver.  Then for each namespace found we check if
> the namespace id and identifiers match to check if we have multiple
> controllers that refer to the same namespaces.  For now path
> availability is based entirely on the controller status, which at
> least for fabrics will be continuously updated based on the mandatory
> keep alive timer.  Once the Asynchronous Namespace Access (ANA)
> proposal passes in NVMe we will also get per-namespace states in
> addition to that, but for now any details of that remain confidential
> to NVMe members.
> 
> The I/O path is very different from the existing multipath drivers,
> which is enabled by the fact that NVMe (unlike SCSI) does not support
> partial completions - a controller will either complete a whole
> command or not, but never only complete parts of it.  Because of that
> there is no need to clone bios or requests - the I/O path simply
> redirects the I/O to a suitable path.  For successful commands
> multipath is not in the completion stack at all.  For failed commands
> we decide if the error could be a path failure, and if yes remove
> the bios from the request structure and requeue them before completing
> the request.  All together this means there is no performance
> degradation compared to normal nvme operation when using the multipath
> device node (at least not until I find a dual ported DRAM backed
> device :))
> 
> A git tree is available at:
> 
>    git://git.infradead.org/users/hch/block.git nvme-mpath
> 
> gitweb:
> 
>    http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/nvme-mpath
> 
> Changes since V3:
>   - new block layer support for hidden gendisks
>   - a couple new patches to refactor device handling before the
>     actual multipath support
>   - don't expose per-controller block device nodes
>   - use /dev/nvmeXnZ as the device nodes for the whole subsystem.

If per-controller block device nodes are hidden, how can the user-space tools
such as multipath-tools and nvme-cli (if it supports) know status of each path of
the multipath device?
In some cases, the admin wants to know which path is in down state , in degraded
state such as suffering intermittent IO error because of shaky link and he can fix
the link or isolate such link from the normal path.

Regards
Guan


>   - expose subsystems in sysfs (Hannes Reinecke)
>   - fix a subsystem leak when duplicate NQNs are found
>   - fix up some names
>   - don't clear current_path if freeing a different namespace
> 
> Changes since V2:
>   - don't create duplicate subsystems on reset (Keith Bush)
>   - free requests properly when failing over in I/O completion (Keith Bush)
>   - new devices names: /dev/nvm-sub%dn%d
>   - expose the namespace identification sysfs files for the mpath nodes
> 
> Changes since V1:
>   - introduce new nvme_ns_ids structure to clean up identifier handling
>   - generic_make_request_fast is now named direct_make_request and calls
>     generic_make_request_checks
>   - reset bi_disk on resubmission
>   - create sysfs links between the existing nvme namespace block devices and
>     the new share mpath device
>   - temporarily added the timeout patches from James, this should go into
>     nvme-4.14, though
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
> 
> .
> 



From sagi at grimberg.me  Sun Oct 22 23:33:35 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 09:33:35 +0300
Subject: nvme multipath support V4
In-Reply-To: <fbccc0db-77cd-fabc-0225-4b515729f214@huawei.com>
References: <20171018165258.23212-1-hch@lst.de>
 <fbccc0db-77cd-fabc-0225-4b515729f214@huawei.com>
Message-ID: <495580b5-2dcf-b547-4c08-8a10fa4844a1@grimberg.me>

Guan,

> If per-controller block device nodes are hidden, how can the user-space tools
> such as multipath-tools and nvme-cli (if it supports) know status of each path of
> the multipath device?

if at all, the path state is reflected on the controller class device
node, not on the namespace block device node. However, this is not
something user-space can rely on as well.

> In some cases, the admin wants to know which path is in down state , in degraded
> state such as suffering intermittent IO error because of shaky link and he can fix
> the link or isolate such link from the normal path.

We have a ctrl_loss_tmo that will remove such a degraded controller
after timeout expiration (default is 600 seconds). When we remove
this controller we log this information so its available for a sysadmin
to fix this if needed/desired.

What we would probably need to do, is either to add a nvme-cli "rescan"
option to discover and connect to all unconnected controllers so the
sysadmin can run after fixing an issue, or introduce a daemon that does
this every say 60 seconds or something.


From jthumshirn at suse.de  Mon Oct 23 01:21:40 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 23 Oct 2017 10:21:40 +0200
Subject: [PATCH v2 1/4] nvme: add duplicate_connect option
In-Reply-To: <20171020231709.3183-2-jsmart2021@gmail.com> (James Smart's
 message of "Fri, 20 Oct 2017 16:17:06 -0700")
References: <20171020231709.3183-1-jsmart2021@gmail.com>
 <20171020231709.3183-2-jsmart2021@gmail.com>
Message-ID: <mqdinf6z33f.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From hch at lst.de  Mon Oct 23 01:23:45 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 10:23:45 +0200
Subject: [PATCH] nvme-rdma: fix possible hang when issuing commands
 during ctrl removal
In-Reply-To: <1508697771-3552-1-git-send-email-sagi@grimberg.me>
References: <1508697771-3552-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171023082345.GA19156@lst.de>

On Sun, Oct 22, 2017 at 09:42:51PM +0300, Sagi Grimberg wrote:
>  			/*
> +			 * deleting state means that the ctrl will never accept
> +			 * commands again, fail it permanently.
> +			 */
> +			if (queue->ctrl->ctrl.state == NVME_CTRL_DELETING) {
> +				nvme_req(rq)->status = NVME_SC_ABORT_REQ;
> +				return BLK_STS_IOERR;

What does the NVME_SC_ABORT_REQ buy us here compared to just returning
BLK_STS_IOERR as in the reconnecting case?  Why can't we just merge the
two cases?


From hch at infradead.org  Mon Oct 23 01:28:05 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 23 Oct 2017 01:28:05 -0700
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171020161253.29296-1-jsmart2021@gmail.com>
References: <20171020161253.29296-1-jsmart2021@gmail.com>
Message-ID: <20171023082805.GA25383@infradead.org>

The whole point of the flush_work suggestion was that we should not
require offloading the deleting to another workqueue for it.


From jthumshirn at suse.de  Mon Oct 23 01:31:00 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 23 Oct 2017 10:31:00 +0200
Subject: [PATCH v2 2/4] nvme: add helper to compare options to controller
In-Reply-To: <20171020231709.3183-3-jsmart2021@gmail.com> (James Smart's
 message of "Fri, 20 Oct 2017 16:17:07 -0700")
References: <20171020231709.3183-1-jsmart2021@gmail.com>
 <20171020231709.3183-3-jsmart2021@gmail.com>
Message-ID: <mqdd15ez2nv.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From sagi at grimberg.me  Mon Oct 23 02:04:18 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 12:04:18 +0300
Subject: [PATCH] nvme-rdma: fix possible hang when issuing commands during
 ctrl removal
In-Reply-To: <20171023082345.GA19156@lst.de>
References: <1508697771-3552-1-git-send-email-sagi@grimberg.me>
 <20171023082345.GA19156@lst.de>
Message-ID: <230d99f3-7f35-5c8d-41b9-5c4867c7f2bb@grimberg.me>


>>   			/*
>> +			 * deleting state means that the ctrl will never accept
>> +			 * commands again, fail it permanently.
>> +			 */
>> +			if (queue->ctrl->ctrl.state == NVME_CTRL_DELETING) {
>> +				nvme_req(rq)->status = NVME_SC_ABORT_REQ;
>> +				return BLK_STS_IOERR;
> 
> What does the NVME_SC_ABORT_REQ buy us here compared to just returning
> BLK_STS_IOERR as in the reconnecting case?

reg_write32() issues a sync command, which returns
nvme_req(req)->status (blk_execute_request error detection is caller
specific). without it nvme_shutdown_ctrl thinks reg_write32() succeeded
and polls with reg_read32() for no good reason.

And its needed for the reconnecting case too, we just never noticed
it for that case.

> Why can't we just merge the two cases?

Because we fail for slightly different reasons so I thought two
distinct conditionals looks cleaner, but we can merge the two.


From jthumshirn at suse.de  Mon Oct 23 02:27:43 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 23 Oct 2017 11:27:43 +0200
Subject: [PATCH v2 3/4] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171020231709.3183-4-jsmart2021@gmail.com> (James Smart's
 message of "Fri, 20 Oct 2017 16:17:08 -0700")
References: <20171020231709.3183-1-jsmart2021@gmail.com>
 <20171020231709.3183-4-jsmart2021@gmail.com>
Message-ID: <mqdbmkyjjsg.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Mon Oct 23 02:28:05 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 23 Oct 2017 11:28:05 +0200
Subject: [PATCH v2 3/4] nvme_fc: add support for duplicate_connect option
In-Reply-To: <20171020231709.3183-4-jsmart2021@gmail.com> (James Smart's
 message of "Fri, 20 Oct 2017 16:17:08 -0700")
References: <20171020231709.3183-1-jsmart2021@gmail.com>
 <20171020231709.3183-4-jsmart2021@gmail.com>
Message-ID: <mqda80ijjru.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jthumshirn at suse.de  Mon Oct 23 02:30:18 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Mon, 23 Oct 2017 11:30:18 +0200
Subject: [PATCH v2 4/4] nvme-rdma: add support for duplicate_connect option
In-Reply-To: <20171020231709.3183-5-jsmart2021@gmail.com> (James Smart's
 message of "Fri, 20 Oct 2017 16:17:09 -0700")
References: <20171020231709.3183-1-jsmart2021@gmail.com>
 <20171020231709.3183-5-jsmart2021@gmail.com>
Message-ID: <mqdwp3mnrdh.fsf@linux-x5ow.site>

Looks good,
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From maxg at mellanox.com  Mon Oct 23 02:59:27 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Mon, 23 Oct 2017 12:59:27 +0300
Subject: [PATCH 1/1] nvme-rdma: align nvme_rdma_device structure
Message-ID: <1508752767-22939-1-git-send-email-maxg@mellanox.com>

Signed-off-by: Max Gurtovoy <maxg at mellanox.com>
---
 drivers/nvme/host/rdma.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..01b9505 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -50,8 +50,8 @@
 	(NVME_AQ_DEPTH - NVME_RDMA_NR_AEN_COMMANDS)
 
 struct nvme_rdma_device {
-	struct ib_device       *dev;
-	struct ib_pd	       *pd;
+	struct ib_device	*dev;
+	struct ib_pd		*pd;
 	struct kref		ref;
 	struct list_head	entry;
 };
-- 
1.7.1



From sagi at grimberg.me  Mon Oct 23 04:05:08 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 14:05:08 +0300
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171023082805.GA25383@infradead.org>
References: <20171020161253.29296-1-jsmart2021@gmail.com>
 <20171023082805.GA25383@infradead.org>
Message-ID: <a1123d71-51f8-1d72-3e5d-67f23634543a@grimberg.me>


> The whole point of the flush_work suggestion was that we should not
> require offloading the deleting to another workqueue for it.

Regardless of flush_work or cancel_work_sync, its a deadlock
in fc.

in rdma/loop we always call ->free_ctrl from a different context.

In rdma we do that from the rdma_cm context, in loop we schedule
host side delete on nvme_wq, in fc apparently we can get to
free_ctrl directly from that context.

If fatal_err_work calls ->delete_ctrl() and that in turn gets to put the
last reference on the ctrl it will end up in ->free_ctrl() under
fatal_err_work context which will then try to flush fatal_err_work.



From sagi at grimberg.me  Mon Oct 23 06:04:11 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 16:04:11 +0300
Subject: [PATCH v2] nvme-rdma: fix possible hang when issuing commands during
 ctrl removal
Message-ID: <1508763851-15338-1-git-send-email-sagi@grimberg.me>

nvme_rdma_queue_is_ready() fails requests in case a queue is not
LIVE. If the controller is in RECONNECTING state, we might be in
this state for a long time (until we successfully reconnect) and
we are better off with failing the request fast. Otherwise, we
fail with BLK_STS_RESOURCE to have the block layer try again
soon.

In case we are removing the controller when the admin queue
is not LIVE, we will terminate the request with BLK_STS_RESOURCE
but it happens before we call blk_mq_start_request() so the
request timeout never expires, and the queue will never get
back to LIVE (because we are removing the controller). This
causes the removal operation to block infinitly [1].

Thus, if we are removing (state DELETING), and the queue is
not LIVE, we need to fail the request permanently as there is
no chance for it to ever complete successfully.

[1]
--
sysrq: SysRq : Show Blocked State
  task                        PC stack   pid father
kworker/u66:2   D    0   440      2 0x80000000
Workqueue: nvme-wq nvme_rdma_del_ctrl_work [nvme_rdma]
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x221/0x580
 io_schedule_timeout+0x1e/0x50
 wait_for_completion_io_timeout+0x118/0x180
 blk_execute_rq+0x86/0xc0
 __nvme_submit_sync_cmd+0x89/0xf0
 nvmf_reg_write32+0x4b/0x90 [nvme_fabrics]
 nvme_shutdown_ctrl+0x41/0xe0
 nvme_rdma_shutdown_ctrl+0xca/0xd0 [nvme_rdma]
 nvme_rdma_remove_ctrl+0x2b/0x40 [nvme_rdma]
 nvme_rdma_del_ctrl_work+0x25/0x30 [nvme_rdma]
 process_one_work+0x1fd/0x630
 worker_thread+0x1db/0x3b0
 kthread+0x11e/0x150
 ret_from_fork+0x27/0x40
01              D    0  2868   2862 0x00000000
Call Trace:
 __schedule+0x3e9/0xb00
 schedule+0x40/0x90
 schedule_timeout+0x260/0x580
 wait_for_completion+0x108/0x170
 flush_work+0x1e0/0x270
 nvme_rdma_del_ctrl+0x5a/0x80 [nvme_rdma]
 nvme_sysfs_delete+0x2a/0x40
 dev_attr_store+0x18/0x30
 sysfs_kf_write+0x45/0x60
 kernfs_fop_write+0x124/0x1c0
 __vfs_write+0x28/0x150
 vfs_write+0xc7/0x1b0
 SyS_write+0x49/0xa0
 entry_SYSCALL_64_fastpath+0x18/0xad
--

Reported-by: Bart Van Assche <bart.vanassche at wdc.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
Changes from v1:
- squashed to ctrl state conditions in nvme_rdma_queue_is_ready

 drivers/nvme/host/rdma.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 8a4a4a5836a8..fd26bcb9e6b7 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1386,12 +1386,15 @@ nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
 			/*
 			 * reconnecting state means transport disruption, which
 			 * can take a long time and even might fail permanently,
-			 * so we can't let incoming I/O be requeued forever.
-			 * fail it fast to allow upper layers a chance to
-			 * failover.
+			 * fail fast to give upper layers a chance to failover.
+			 * deleting state means that the ctrl will never accept
+			 * commands again, fail it permanently.
 			 */
-			if (queue->ctrl->ctrl.state == NVME_CTRL_RECONNECTING)
+			if (queue->ctrl->ctrl.state == NVME_CTRL_RECONNECTING ||
+			    queue->ctrl->ctrl.state == NVME_CTRL_DELETING) {
+				nvme_req(rq)->status = NVME_SC_ABORT_REQ;
 				return BLK_STS_IOERR;
+			}
 			return BLK_STS_RESOURCE; /* try again later */
 		}
 	}
-- 
2.7.4



From keith.busch at intel.com  Mon Oct 23 07:24:39 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 23 Oct 2017 08:24:39 -0600
Subject: [PATCH] nvme-cli: make address print format more human-readable
In-Reply-To: <1508559796-7605-1-git-send-email-minwoo.im.dev@gmail.com>
References: <1508559796-7605-1-git-send-email-minwoo.im.dev@gmail.com>
Message-ID: <20171023142438.GB30713@localhost.localdomain>

On Sat, Oct 21, 2017 at 01:23:16PM +0900, Minwoo Im wrote:
> Memory address would be great in hexadecimal format to be read easily.
> 
> Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>

Applied, thanks.


From hch at lst.de  Mon Oct 23 07:25:44 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:25:44 +0200
Subject: [PATCH v2] nvme-rdma: fix possible hang when issuing commands
 during  ctrl removal
In-Reply-To: <1508763851-15338-1-git-send-email-sagi@grimberg.me>
References: <1508763851-15338-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171023142544.GA25316@lst.de>

Looks fine, I'll queue it up for 4.14.


From hch at lst.de  Mon Oct 23 07:26:37 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:26:37 +0200
Subject: [PATCH 1/1] nvme-rdma: align nvme_rdma_device structure
In-Reply-To: <1508752767-22939-1-git-send-email-maxg@mellanox.com>
References: <1508752767-22939-1-git-send-email-maxg@mellanox.com>
Message-ID: <20171023142637.GB25316@lst.de>

On Mon, Oct 23, 2017 at 12:59:27PM +0300, Max Gurtovoy wrote:
>  struct nvme_rdma_device {
> -	struct ib_device       *dev;
> -	struct ib_pd	       *pd;
> +	struct ib_device	*dev;
> +	struct ib_pd		*pd;

Looks fine, I'll queue it up for 4.15.


From hch at lst.de  Mon Oct 23 07:27:24 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:27:24 +0200
Subject: [PATCH] nvme-rdma: Add debug message when reaches timeout
In-Reply-To: <20171022093704.25203-1-nitzanc@mellanox.com>
References: <20171022093704.25203-1-nitzanc@mellanox.com>
Message-ID: <20171023142724.GC25316@lst.de>

I'll apply it for 4.15 after fixing up the message to really match
PCI..


From hch at lst.de  Mon Oct 23 07:31:42 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:31:42 +0200
Subject: [PATCH 1/5] nvme-core: Make nvme_init/uninit_ctrl
 setup/teardown symmetric
In-Reply-To: <d1e113b8-1f76-1084-b075-a32d484287d8@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-2-git-send-email-sagi@grimberg.me>
 <20171019152910.GA24808@lst.de>
 <0a081ffc-5a3d-6b52-7257-18e39034ce93@grimberg.me>
 <d1e113b8-1f76-1084-b075-a32d484287d8@grimberg.me>
Message-ID: <20171023143142.GA25448@lst.de>

On Sun, Oct 22, 2017 at 02:11:47PM +0300, Sagi Grimberg wrote:
>>> This totally conflicts with my refcount / device rework in the
>>> mpath patchset..
>>
>> I know...
>>
>> Do you want me to rebase on top of your next round of patches?
>
> ping?

Sure.  It might also make sense to queue up the device / refcount
updates from th mpath series to nvme-4.15 ASAP so you only have
to base on top of those.


From hch at lst.de  Mon Oct 23 07:32:55 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:32:55 +0200
Subject: [PATCH 2/5] nvme-rdma: rework create_ctrl error flow
In-Reply-To: <1508324864-23087-3-git-send-email-sagi@grimberg.me>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-3-git-send-email-sagi@grimberg.me>
Message-ID: <20171023143255.GB25448@lst.de>

On Wed, Oct 18, 2017 at 02:07:41PM +0300, Sagi Grimberg wrote:
> From: Roy Shterman <roys at lightbitslabs.com>
> 
> The magic ctrl refcount handling we have due to the asymmetric
> behavior in nvme_init/uninit_ctrl is making the controller
> creation error sequence a bit awkward and error prone as we
> rely on magic refcounting to invoke ->free_ctrl from create_ctrl
> error sequence.
> 
> Now that we have nvme_init/uninit_ctrl doing the right thing, we
> can fix this by making the error sequence sane and simple (just
> undo what we did until the error).
> 
> Also, allocate ctrl->queues before calling nvme_init_ctrl, just
> to separate the driver private resource allocation and the core
> resource allocations.

Once we did the init we'll have to do a put and not just free it.
That will become even more important when using a struct device for
refcounting the controller.


From maxg at mellanox.com  Mon Oct 23 07:42:55 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Mon, 23 Oct 2017 17:42:55 +0300
Subject: [PATCH v2] nvme-rdma: fix possible hang when issuing commands
 during ctrl removal
In-Reply-To: <1508763851-15338-1-git-send-email-sagi@grimberg.me>
References: <1508763851-15338-1-git-send-email-sagi@grimberg.me>
Message-ID: <1bfc4bbb-b841-c547-8c08-c259996c7e59@mellanox.com>


> 
> Reported-by: Bart Van Assche <bart.vanassche at wdc.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
> Changes from v1:
> - squashed to ctrl state conditions in nvme_rdma_queue_is_ready
> 
>   drivers/nvme/host/rdma.c | 11 +++++++----
>   1 file changed, 7 insertions(+), 4 deletions(-)
> 

Looks good,
Reviewed-by: Max Gurtovoy <maxg at mellanox.com>


From hch at infradead.org  Mon Oct 23 07:50:13 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Mon, 23 Oct 2017 07:50:13 -0700
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <a1123d71-51f8-1d72-3e5d-67f23634543a@grimberg.me>
References: <20171020161253.29296-1-jsmart2021@gmail.com>
 <20171023082805.GA25383@infradead.org>
 <a1123d71-51f8-1d72-3e5d-67f23634543a@grimberg.me>
Message-ID: <20171023145013.GA3017@infradead.org>

On Mon, Oct 23, 2017 at 02:05:08PM +0300, Sagi Grimberg wrote:
> Regardless of flush_work or cancel_work_sync, its a deadlock
> in fc.
> 
> in rdma/loop we always call ->free_ctrl from a different context.
> 
> In rdma we do that from the rdma_cm context, in loop we schedule
> host side delete on nvme_wq, in fc apparently we can get to
> free_ctrl directly from that context.

Yes, nvmet_fc_delete_ctrl -> nvmet_fc_delete_target_assoc ->
nvmet_fc_delete_target_queue.

> If fatal_err_work calls ->delete_ctrl() and that in turn gets to put the
> last reference on the ctrl it will end up in ->free_ctrl() under
> fatal_err_work context which will then try to flush fatal_err_work.

Yes, and the way I understand flush_work that is perfectly fine for
flush_work, just not for cancel_work_sync.


From hch at lst.de  Mon Oct 23 07:51:09 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:09 +0200
Subject: nvme multipath support V5
Message-ID: <20171023145126.2471-1-hch@lst.de>

Hi all,

this series adds support for multipathing, that is accessing nvme
namespaces through multiple controllers to the nvme core driver.

It is a very thin and efficient implementation that relies on
close cooperation with other bits of the nvme driver, and few small
and simple block helpers.

Compared to dm-multipath the important differences are how management
of the paths is done, and how the I/O path works.

Management of the paths is fully integrated into the nvme driver,
for each newly found nvme controller we check if there are other
controllers that refer to the same subsystem, and if so we link them
up in the nvme driver.  Then for each namespace found we check if
the namespace id and identifiers match to check if we have multiple
controllers that refer to the same namespaces.  For now path
availability is based entirely on the controller status, which at
least for fabrics will be continuously updated based on the mandatory
keep alive timer.  Once the Asynchronous Namespace Access (ANA)
proposal passes in NVMe we will also get per-namespace states in
addition to that, but for now any details of that remain confidential
to NVMe members.

The I/O path is very different from the existing multipath drivers,
which is enabled by the fact that NVMe (unlike SCSI) does not support
partial completions - a controller will either complete a whole
command or not, but never only complete parts of it.  Because of that
there is no need to clone bios or requests - the I/O path simply
redirects the I/O to a suitable path.  For successful commands
multipath is not in the completion stack at all.  For failed commands
we decide if the error could be a path failure, and if yes remove
the bios from the request structure and requeue them before completing
the request.  All together this means there is no performance
degradation compared to normal nvme operation when using the multipath
device node (at least not until I find a dual ported DRAM backed
device :))

A git tree is available at:

   git://git.infradead.org/users/hch/block.git nvme-mpath

gitweb:

   http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/nvme-mpath

Changes since V4:
  - add a refcount to release the device in struct nvme_subsystem
  - use the instance to name the nvme_subsystems in sysfs
  - remove a NULL check before nvme_put_ns_head
  - take a ns_head reference in ->open
  - improve open protection for GENHD_FL_HIDDEN
  - add poll support for the mpath device

Changes since V3:
  - new block layer support for hidden gendisks
  - a couple new patches to refactor device handling before the
    actual multipath support
  - don't expose per-controller block device nodes
  - use /dev/nvmeXnZ as the device nodes for the whole subsystem.
  - expose subsystems in sysfs (Hannes Reinecke)
  - fix a subsystem leak when duplicate NQNs are found
  - fix up some names
  - don't clear current_path if freeing a different namespace

Changes since V2:
  - don't create duplicate subsystems on reset (Keith Bush)
  - free requests properly when failing over in I/O completion (Keith Bush)
  - new devices names: /dev/nvm-sub%dn%d
  - expose the namespace identification sysfs files for the mpath nodes

Changes since V1:
  - introduce new nvme_ns_ids structure to clean up identifier handling
  - generic_make_request_fast is now named direct_make_request and calls
    generic_make_request_checks
  - reset bi_disk on resubmission
  - create sysfs links between the existing nvme namespace block devices and
    the new share mpath device
  - temporarily added the timeout patches from James, this should go into
    nvme-4.14, though


From hch at lst.de  Mon Oct 23 07:51:10 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:10 +0200
Subject: [PATCH 01/17] block: move REQ_NOWAIT
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-2-hch@lst.de>

This flag should be before the operation-specific REQ_NOUNMAP bit.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Hannes Reinecke <hare at suse.com>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 include/linux/blk_types.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index a2d2aa709cef..acc2f3cdc2fc 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -224,11 +224,11 @@ enum req_flag_bits {
 	__REQ_PREFLUSH,		/* request for cache flush */
 	__REQ_RAHEAD,		/* read ahead, can fail anytime */
 	__REQ_BACKGROUND,	/* background IO */
+	__REQ_NOWAIT,           /* Don't wait if request will block */
 
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
-	__REQ_NOWAIT,           /* Don't wait if request will block */
 	__REQ_NR_BITS,		/* stops here */
 };
 
@@ -245,9 +245,9 @@ enum req_flag_bits {
 #define REQ_PREFLUSH		(1ULL << __REQ_PREFLUSH)
 #define REQ_RAHEAD		(1ULL << __REQ_RAHEAD)
 #define REQ_BACKGROUND		(1ULL << __REQ_BACKGROUND)
+#define REQ_NOWAIT		(1ULL << __REQ_NOWAIT)
 
 #define REQ_NOUNMAP		(1ULL << __REQ_NOUNMAP)
-#define REQ_NOWAIT		(1ULL << __REQ_NOWAIT)
 
 #define REQ_FAILFAST_MASK \
 	(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:11 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:11 +0200
Subject: [PATCH 02/17] block: add REQ_DRV bit
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-3-hch@lst.de>

Set aside a bit in the request/bio flags for driver use.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Hannes Reinecke <hare at suse.com>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 include/linux/blk_types.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index acc2f3cdc2fc..7ec2ed097a8a 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -229,6 +229,9 @@ enum req_flag_bits {
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
+	/* for driver use */
+	__REQ_DRV,
+
 	__REQ_NR_BITS,		/* stops here */
 };
 
@@ -249,6 +252,8 @@ enum req_flag_bits {
 
 #define REQ_NOUNMAP		(1ULL << __REQ_NOUNMAP)
 
+#define REQ_DRV			(1ULL << __REQ_DRV)
+
 #define REQ_FAILFAST_MASK \
 	(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
 
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:12 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:12 +0200
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-4-hch@lst.de>

This helper allows reinserting a bio into a new queue without much
overhead, but requires all queue limits to be the same for the upper
and lower queues, and it does not provide any recursion preventions.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 block/blk-core.c       | 34 ++++++++++++++++++++++++++++++++++
 include/linux/blkdev.h |  1 +
 2 files changed, 35 insertions(+)

diff --git a/block/blk-core.c b/block/blk-core.c
index 14f7674fa0b1..b8c80f39f5fe 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2241,6 +2241,40 @@ blk_qc_t generic_make_request(struct bio *bio)
 }
 EXPORT_SYMBOL(generic_make_request);
 
+/**
+ * direct_make_request - hand a buffer directly to its device driver for I/O
+ * @bio:  The bio describing the location in memory and on the device.
+ *
+ * This function behaves like generic_make_request(), but does not protect
+ * against recursion.  Must only be used if the called driver is known
+ * to not call generic_make_request (or direct_make_request) again from
+ * its make_request function.  (Calling direct_make_request again from
+ * a workqueue is perfectly fine as that doesn't recurse).
+ */
+blk_qc_t direct_make_request(struct bio *bio)
+{
+	struct request_queue *q = bio->bi_disk->queue;
+	bool nowait = bio->bi_opf & REQ_NOWAIT;
+	blk_qc_t ret;
+
+	if (!generic_make_request_checks(bio))
+		return BLK_QC_T_NONE;
+
+	if (unlikely(blk_queue_enter(q, nowait))) {
+		if (nowait && !blk_queue_dying(q))
+			bio->bi_status = BLK_STS_AGAIN;
+		else
+			bio->bi_status = BLK_STS_IOERR;
+		bio_endio(bio);
+		return BLK_QC_T_NONE;
+	}
+
+	ret = q->make_request_fn(q, bio);
+	blk_queue_exit(q);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(direct_make_request);
+
 /**
  * submit_bio - submit a bio to the block device layer for I/O
  * @bio: The &struct bio which describes the I/O
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 02fa42d24b52..780f01db5899 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -936,6 +936,7 @@ do {								\
 extern int blk_register_queue(struct gendisk *disk);
 extern void blk_unregister_queue(struct gendisk *disk);
 extern blk_qc_t generic_make_request(struct bio *bio);
+extern blk_qc_t direct_make_request(struct bio *bio);
 extern void blk_rq_init(struct request_queue *q, struct request *rq);
 extern void blk_init_request_from_bio(struct request *req, struct bio *bio);
 extern void blk_put_request(struct request *);
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:13 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:13 +0200
Subject: [PATCH 04/17] block: add a blk_steal_bios helper
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-5-hch@lst.de>

This helpers allows to bounce steal the uncompleted bios from a request so
that they can be reissued on another path.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 block/blk-core.c       | 20 ++++++++++++++++++++
 include/linux/blkdev.h |  2 ++
 2 files changed, 22 insertions(+)

diff --git a/block/blk-core.c b/block/blk-core.c
index b8c80f39f5fe..e804529e65a5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2767,6 +2767,26 @@ struct request *blk_fetch_request(struct request_queue *q)
 }
 EXPORT_SYMBOL(blk_fetch_request);
 
+/*
+ * Steal bios from a request.  The request must not have been partially
+ * completed before.
+ */
+void blk_steal_bios(struct bio_list *list, struct request *rq)
+{
+	if (rq->bio) {
+		if (list->tail)
+			list->tail->bi_next = rq->bio;
+		else
+			list->head = rq->bio;
+		list->tail = rq->biotail;
+	}
+
+	rq->bio = NULL;
+	rq->biotail = NULL;
+	rq->__data_len = 0;
+}
+EXPORT_SYMBOL_GPL(blk_steal_bios);
+
 /**
  * blk_update_request - Special helper function for request stacking drivers
  * @req:      the request being processed
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 780f01db5899..45c63764a14e 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1110,6 +1110,8 @@ extern struct request *blk_peek_request(struct request_queue *q);
 extern void blk_start_request(struct request *rq);
 extern struct request *blk_fetch_request(struct request_queue *q);
 
+void blk_steal_bios(struct bio_list *list, struct request *rq);
+
 /*
  * Request completion related functions.
  *
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:16 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:16 +0200
Subject: [PATCH 07/17] block: add a poll_fn callback to struct request_queue
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-8-hch@lst.de>

That we we can also poll non blk-mq queues.  Mostly needed for
the NVMe multipath code, but could also be useful elsewhere.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 block/blk-core.c             | 11 +++++++++++
 block/blk-mq.c               | 14 +++++---------
 drivers/nvme/target/io-cmd.c |  2 +-
 fs/block_dev.c               |  4 ++--
 fs/direct-io.c               |  2 +-
 fs/iomap.c                   |  2 +-
 include/linux/blkdev.h       |  4 +++-
 mm/page_io.c                 |  2 +-
 8 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index e804529e65a5..8e7e12e5ffa2 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2319,6 +2319,17 @@ blk_qc_t submit_bio(struct bio *bio)
 }
 EXPORT_SYMBOL(submit_bio);
 
+bool blk_poll(struct request_queue *q, blk_qc_t cookie)
+{
+	if (!q->poll_fn || !blk_qc_t_valid(cookie))
+		return false;
+
+	if (current->plug)
+		blk_flush_plug_list(current->plug, false);
+	return q->poll_fn(q, cookie);
+}
+EXPORT_SYMBOL_GPL(blk_poll);
+
 /**
  * blk_cloned_rq_check_limits - Helper function to check a cloned request
  *                              for new the queue limits
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 7f01d69879d6..10c99cf6fd71 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -37,6 +37,7 @@
 #include "blk-wbt.h"
 #include "blk-mq-sched.h"
 
+static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie);
 static void blk_mq_poll_stats_start(struct request_queue *q);
 static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb);
 
@@ -2401,6 +2402,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	spin_lock_init(&q->requeue_lock);
 
 	blk_queue_make_request(q, blk_mq_make_request);
+	if (q->mq_ops->poll)
+		q->poll_fn = blk_mq_poll;
 
 	/*
 	 * Do this after blk_queue_make_request() overrides it...
@@ -2860,20 +2863,14 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq)
 	return false;
 }
 
-bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie)
+static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie)
 {
 	struct blk_mq_hw_ctx *hctx;
-	struct blk_plug *plug;
 	struct request *rq;
 
-	if (!q->mq_ops || !q->mq_ops->poll || !blk_qc_t_valid(cookie) ||
-	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
+	if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
 		return false;
 
-	plug = current->plug;
-	if (plug)
-		blk_flush_plug_list(plug, false);
-
 	hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)];
 	if (!blk_qc_t_is_internal(cookie))
 		rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie));
@@ -2891,7 +2888,6 @@ bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie)
 
 	return __blk_mq_poll(hctx, rq);
 }
-EXPORT_SYMBOL_GPL(blk_mq_poll);
 
 static int __init blk_mq_init(void)
 {
diff --git a/drivers/nvme/target/io-cmd.c b/drivers/nvme/target/io-cmd.c
index 0d4c23dc4532..db632818777d 100644
--- a/drivers/nvme/target/io-cmd.c
+++ b/drivers/nvme/target/io-cmd.c
@@ -94,7 +94,7 @@ static void nvmet_execute_rw(struct nvmet_req *req)
 
 	cookie = submit_bio(bio);
 
-	blk_mq_poll(bdev_get_queue(req->ns->bdev), cookie);
+	blk_poll(bdev_get_queue(req->ns->bdev), cookie);
 }
 
 static void nvmet_execute_flush(struct nvmet_req *req)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 93d088ffc05c..49a55246ba50 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -249,7 +249,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
 		if (!READ_ONCE(bio.bi_private))
 			break;
 		if (!(iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_mq_poll(bdev_get_queue(bdev), qc))
+		    !blk_poll(bdev_get_queue(bdev), qc))
 			io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
@@ -414,7 +414,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
 			break;
 
 		if (!(iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_mq_poll(bdev_get_queue(bdev), qc))
+		    !blk_poll(bdev_get_queue(bdev), qc))
 			io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
diff --git a/fs/direct-io.c b/fs/direct-io.c
index 62cf812ed0e5..d2bc339cb1e9 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -486,7 +486,7 @@ static struct bio *dio_await_one(struct dio *dio)
 		dio->waiter = current;
 		spin_unlock_irqrestore(&dio->bio_lock, flags);
 		if (!(dio->iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_mq_poll(dio->bio_disk->queue, dio->bio_cookie))
+		    !blk_poll(dio->bio_disk->queue, dio->bio_cookie))
 			io_schedule();
 		/* wake up sets us TASK_RUNNING */
 		spin_lock_irqsave(&dio->bio_lock, flags);
diff --git a/fs/iomap.c b/fs/iomap.c
index 8194d30bdca0..4241bac905b1 100644
--- a/fs/iomap.c
+++ b/fs/iomap.c
@@ -1049,7 +1049,7 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 
 			if (!(iocb->ki_flags & IOCB_HIPRI) ||
 			    !dio->submit.last_queue ||
-			    !blk_mq_poll(dio->submit.last_queue,
+			    !blk_poll(dio->submit.last_queue,
 					 dio->submit.cookie))
 				io_schedule();
 		}
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 45c63764a14e..a18ea9b9b8f7 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -266,6 +266,7 @@ struct blk_queue_ctx;
 
 typedef void (request_fn_proc) (struct request_queue *q);
 typedef blk_qc_t (make_request_fn) (struct request_queue *q, struct bio *bio);
+typedef bool (poll_q_fn) (struct request_queue *q, blk_qc_t);
 typedef int (prep_rq_fn) (struct request_queue *, struct request *);
 typedef void (unprep_rq_fn) (struct request_queue *, struct request *);
 
@@ -408,6 +409,7 @@ struct request_queue {
 
 	request_fn_proc		*request_fn;
 	make_request_fn		*make_request_fn;
+	poll_q_fn		*poll_fn;
 	prep_rq_fn		*prep_rq_fn;
 	unprep_rq_fn		*unprep_rq_fn;
 	softirq_done_fn		*softirq_done_fn;
@@ -991,7 +993,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *,
 int blk_status_to_errno(blk_status_t status);
 blk_status_t errno_to_blk_status(int errno);
 
-bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie);
+bool blk_poll(struct request_queue *q, blk_qc_t cookie);
 
 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
 {
diff --git a/mm/page_io.c b/mm/page_io.c
index 21502d341a67..ff04de630c46 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -407,7 +407,7 @@ int swap_readpage(struct page *page, bool do_poll)
 		if (!READ_ONCE(bio->bi_private))
 			break;
 
-		if (!blk_mq_poll(disk->queue, qc))
+		if (!blk_poll(disk->queue, qc))
 			break;
 	}
 	__set_current_state(TASK_RUNNING);
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:14 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:14 +0200
Subject: [PATCH 05/17] block: don't look at the struct device dev_t in
 disk_devt
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-6-hch@lst.de>

The hidden gendisks introduced in the next patch need to keep the dev
field in their struct device empty so that udev won't try to create
block device nodes for them.  To support that rewrite disk_devt to
look at the major and first_minor fields in the gendisk itself instead
of looking into the struct device.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 block/genhd.c         | 4 ----
 include/linux/genhd.h | 2 +-
 2 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index dd305c65ffb0..1174d24e405e 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -649,10 +649,6 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 		return;
 	}
 	disk_to_dev(disk)->devt = devt;
-
-	/* ->major and ->first_minor aren't supposed to be
-	 * dereferenced from here on, but set them just in case.
-	 */
 	disk->major = MAJOR(devt);
 	disk->first_minor = MINOR(devt);
 
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index ea652bfcd675..5c0ed5db33c2 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -234,7 +234,7 @@ static inline bool disk_part_scan_enabled(struct gendisk *disk)
 
 static inline dev_t disk_devt(struct gendisk *disk)
 {
-	return disk_to_dev(disk)->devt;
+	return MKDEV(disk->major, disk->first_minor);
 }
 
 static inline dev_t part_devt(struct hd_struct *part)
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:15 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:15 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-7-hch@lst.de>

With this flag a driver can create a gendisk that can be used for I/O
submission inside the kernel, but which is not registered as user
facing block device.  This will be useful for the NVMe multipath
implementation.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 block/genhd.c         | 57 +++++++++++++++++++++++++++++++++++----------------
 include/linux/genhd.h |  1 +
 2 files changed, 40 insertions(+), 18 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index 1174d24e405e..11a41cca3475 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -585,6 +585,11 @@ static void register_disk(struct device *parent, struct gendisk *disk)
 	 */
 	pm_runtime_set_memalloc_noio(ddev, true);
 
+	if (disk->flags & GENHD_FL_HIDDEN) {
+		dev_set_uevent_suppress(ddev, 0);
+		return;
+	}
+
 	disk->part0.holder_dir = kobject_create_and_add("holders", &ddev->kobj);
 	disk->slave_dir = kobject_create_and_add("slaves", &ddev->kobj);
 
@@ -616,6 +621,11 @@ static void register_disk(struct device *parent, struct gendisk *disk)
 	while ((part = disk_part_iter_next(&piter)))
 		kobject_uevent(&part_to_dev(part)->kobj, KOBJ_ADD);
 	disk_part_iter_exit(&piter);
+
+	err = sysfs_create_link(&ddev->kobj,
+				&disk->queue->backing_dev_info->dev->kobj,
+				"bdi");
+	WARN_ON(err);
 }
 
 /**
@@ -630,7 +640,6 @@ static void register_disk(struct device *parent, struct gendisk *disk)
  */
 void device_add_disk(struct device *parent, struct gendisk *disk)
 {
-	struct backing_dev_info *bdi;
 	dev_t devt;
 	int retval;
 
@@ -639,7 +648,8 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 	 * parameters make sense.
 	 */
 	WARN_ON(disk->minors && !(disk->major || disk->first_minor));
-	WARN_ON(!disk->minors && !(disk->flags & GENHD_FL_EXT_DEVT));
+	WARN_ON(!disk->minors &&
+		!(disk->flags & (GENHD_FL_EXT_DEVT | GENHD_FL_HIDDEN)));
 
 	disk->flags |= GENHD_FL_UP;
 
@@ -648,18 +658,26 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 		WARN_ON(1);
 		return;
 	}
-	disk_to_dev(disk)->devt = devt;
 	disk->major = MAJOR(devt);
 	disk->first_minor = MINOR(devt);
 
 	disk_alloc_events(disk);
 
-	/* Register BDI before referencing it from bdev */
-	bdi = disk->queue->backing_dev_info;
-	bdi_register_owner(bdi, disk_to_dev(disk));
-
-	blk_register_region(disk_devt(disk), disk->minors, NULL,
-			    exact_match, exact_lock, disk);
+	if (disk->flags & GENHD_FL_HIDDEN) {
+		/*
+		 * Don't let hidden disks show up in /proc/partitions,
+		 * and don't bother scanning for partitions either.
+		 */
+		disk->flags |= GENHD_FL_SUPPRESS_PARTITION_INFO;
+		disk->flags |= GENHD_FL_NO_PART_SCAN;
+	} else {
+		/* Register BDI before referencing it from bdev */
+		disk_to_dev(disk)->devt = devt;
+		bdi_register_owner(disk->queue->backing_dev_info,
+				disk_to_dev(disk));
+		blk_register_region(disk_devt(disk), disk->minors, NULL,
+				    exact_match, exact_lock, disk);
+	}
 	register_disk(parent, disk);
 	blk_register_queue(disk);
 
@@ -669,10 +687,6 @@ void device_add_disk(struct device *parent, struct gendisk *disk)
 	 */
 	WARN_ON_ONCE(!blk_get_queue(disk->queue));
 
-	retval = sysfs_create_link(&disk_to_dev(disk)->kobj, &bdi->dev->kobj,
-				   "bdi");
-	WARN_ON(retval);
-
 	disk_add_events(disk);
 	blk_integrity_add(disk);
 }
@@ -701,7 +715,8 @@ void del_gendisk(struct gendisk *disk)
 	set_capacity(disk, 0);
 	disk->flags &= ~GENHD_FL_UP;
 
-	sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
+	if (!(disk->flags & GENHD_FL_HIDDEN))
+		sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
 	if (disk->queue) {
 		/*
 		 * Unregister bdi before releasing device numbers (as they can
@@ -712,13 +727,15 @@ void del_gendisk(struct gendisk *disk)
 	} else {
 		WARN_ON(1);
 	}
-	blk_unregister_region(disk_devt(disk), disk->minors);
+
+	if (!(disk->flags & GENHD_FL_HIDDEN)) {
+		blk_unregister_region(disk_devt(disk), disk->minors);
+		kobject_put(disk->part0.holder_dir);
+		kobject_put(disk->slave_dir);
+	}
 
 	part_stat_set_all(&disk->part0, 0);
 	disk->part0.stamp = 0;
-
-	kobject_put(disk->part0.holder_dir);
-	kobject_put(disk->slave_dir);
 	if (!sysfs_deprecated)
 		sysfs_remove_link(block_depr, dev_name(disk_to_dev(disk)));
 	pm_runtime_set_memalloc_noio(disk_to_dev(disk), false);
@@ -781,6 +798,10 @@ struct gendisk *get_gendisk(dev_t devt, int *partno)
 		spin_unlock_bh(&ext_devt_lock);
 	}
 
+	if (unlikely(disk->flags & GENHD_FL_HIDDEN)) {
+		put_disk(disk);
+		disk = NULL;
+	}
 	return disk;
 }
 EXPORT_SYMBOL(get_gendisk);
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 5c0ed5db33c2..93aae3476f58 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -140,6 +140,7 @@ struct hd_struct {
 #define GENHD_FL_NATIVE_CAPACITY		128
 #define GENHD_FL_BLOCK_EVENTS_ON_EXCL_WRITE	256
 #define GENHD_FL_NO_PART_SCAN			512
+#define GENHD_FL_HIDDEN				1024
 
 enum {
 	DISK_EVENT_MEDIA_CHANGE			= 1 << 0, /* media changed */
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:17 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:17 +0200
Subject: [PATCH 08/17] nvme: use kref_get_unless_zero in nvme_find_get_ns
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-9-hch@lst.de>

For kref_get_unless_zero to protect against lookup vs free races we need
to use it in all places where we aren't guaranteed to already hold a
reference.  There is no such guarantee in nvme_find_get_ns, so switch to
kref_get_unless_zero in this function.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7fae42d595d5..1d931deac83b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2290,7 +2290,8 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	mutex_lock(&ctrl->namespaces_mutex);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
 		if (ns->ns_id == nsid) {
-			kref_get(&ns->kref);
+			if (!kref_get_unless_zero(&ns->kref))
+				continue;
 			ret = ns;
 			break;
 		}
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:18 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:18 +0200
Subject: [PATCH 09/17] nvme: simplify nvme_open
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-10-hch@lst.de>

Now that we are protected against lookup vs free races for the namespace
by using kref_get_unless_zero we don't need the hack of NULLing out the
disk private data during removal.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 40 ++++++++++------------------------------
 1 file changed, 10 insertions(+), 30 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1d931deac83b..9f8ae15c9fe8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -253,12 +253,6 @@ static void nvme_free_ns(struct kref *kref)
 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
 
-	if (ns->disk) {
-		spin_lock(&dev_list_lock);
-		ns->disk->private_data = NULL;
-		spin_unlock(&dev_list_lock);
-	}
-
 	put_disk(ns->disk);
 	ida_simple_remove(&ns->ctrl->ns_ida, ns->instance);
 	nvme_put_ctrl(ns->ctrl);
@@ -270,29 +264,6 @@ static void nvme_put_ns(struct nvme_ns *ns)
 	kref_put(&ns->kref, nvme_free_ns);
 }
 
-static struct nvme_ns *nvme_get_ns_from_disk(struct gendisk *disk)
-{
-	struct nvme_ns *ns;
-
-	spin_lock(&dev_list_lock);
-	ns = disk->private_data;
-	if (ns) {
-		if (!kref_get_unless_zero(&ns->kref))
-			goto fail;
-		if (!try_module_get(ns->ctrl->ops->module))
-			goto fail_put_ns;
-	}
-	spin_unlock(&dev_list_lock);
-
-	return ns;
-
-fail_put_ns:
-	kref_put(&ns->kref, nvme_free_ns);
-fail:
-	spin_unlock(&dev_list_lock);
-	return NULL;
-}
-
 struct request *nvme_alloc_request(struct request_queue *q,
 		struct nvme_command *cmd, unsigned int flags, int qid)
 {
@@ -1056,7 +1027,16 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 
 static int nvme_open(struct block_device *bdev, fmode_t mode)
 {
-	return nvme_get_ns_from_disk(bdev->bd_disk) ? 0 : -ENXIO;
+	struct nvme_ns *ns = bdev->bd_disk->private_data;
+
+	if (!kref_get_unless_zero(&ns->kref))
+		return -ENXIO;
+	if (!try_module_get(ns->ctrl->ops->module)) {
+		kref_put(&ns->kref, nvme_free_ns);
+		return -ENXIO;
+	}
+
+	return 0;
 }
 
 static void nvme_release(struct gendisk *disk, fmode_t mode)
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:19 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:19 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct device
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-11-hch@lst.de>

Instead of allocating a separate struct device for the character device
handle embedd it into struct nvme_ctrl and use it for the main controller
refcounting.  This removes double refcounting and gets us an automatic
reference for the character device operations.  We keep ctrl->device as a
pointer for now to avoid chaning printks all over, but in the future we
could look into message printing helpers that take a controller structure
similar to what other subsystems do.

Note the delete_ctrl operation always already has a reference (either
through sysfs due this change, or because every open file on the
/dev/nvme-fabrics node has a refernece) when it is entered now, so we
don't need to do the unless_zero variant there.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c   | 43 ++++++++++++++++++++++---------------------
 drivers/nvme/host/fc.c     |  8 ++------
 drivers/nvme/host/nvme.h   | 12 +++++++++++-
 drivers/nvme/host/pci.c    |  2 +-
 drivers/nvme/host/rdma.c   |  5 ++---
 drivers/nvme/target/loop.c |  2 +-
 6 files changed, 39 insertions(+), 33 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 9f8ae15c9fe8..3a97daa163f6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1915,7 +1915,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file)
 			ret = -EWOULDBLOCK;
 			break;
 		}
-		if (!kref_get_unless_zero(&ctrl->kref))
+		if (!kobject_get_unless_zero(&ctrl->device->kobj))
 			break;
 		file->private_data = ctrl;
 		ret = 0;
@@ -2374,7 +2374,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	list_add_tail(&ns->list, &ctrl->namespaces);
 	mutex_unlock(&ctrl->namespaces_mutex);
 
-	kref_get(&ctrl->kref);
+	nvme_get_ctrl(ctrl);
 
 	kfree(id);
 
@@ -2703,7 +2703,7 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
-	device_destroy(nvme_class, MKDEV(nvme_char_major, ctrl->instance));
+	device_del(ctrl->device);
 
 	spin_lock(&dev_list_lock);
 	list_del(&ctrl->node);
@@ -2711,23 +2711,17 @@ void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
 
-static void nvme_free_ctrl(struct kref *kref)
+static void nvme_free_ctrl(struct device *dev)
 {
-	struct nvme_ctrl *ctrl = container_of(kref, struct nvme_ctrl, kref);
+	struct nvme_ctrl *ctrl =
+		container_of(dev, struct nvme_ctrl, ctrl_device);
 
-	put_device(ctrl->device);
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 	ida_destroy(&ctrl->ns_ida);
 
 	ctrl->ops->free_ctrl(ctrl);
 }
 
-void nvme_put_ctrl(struct nvme_ctrl *ctrl)
-{
-	kref_put(&ctrl->kref, nvme_free_ctrl);
-}
-EXPORT_SYMBOL_GPL(nvme_put_ctrl);
-
 /*
  * Initialize a NVMe controller structures.  This needs to be called during
  * earliest initialization so that we have the initialized structured around
@@ -2742,7 +2736,6 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	spin_lock_init(&ctrl->lock);
 	INIT_LIST_HEAD(&ctrl->namespaces);
 	mutex_init(&ctrl->namespaces_mutex);
-	kref_init(&ctrl->kref);
 	ctrl->dev = dev;
 	ctrl->ops = ops;
 	ctrl->quirks = quirks;
@@ -2755,15 +2748,21 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 		goto out;
 	ctrl->instance = ret;
 
-	ctrl->device = device_create_with_groups(nvme_class, ctrl->dev,
-				MKDEV(nvme_char_major, ctrl->instance),
-				ctrl, nvme_dev_attr_groups,
-				"nvme%d", ctrl->instance);
-	if (IS_ERR(ctrl->device)) {
-		ret = PTR_ERR(ctrl->device);
+	device_initialize(&ctrl->ctrl_device);
+	ctrl->device = &ctrl->ctrl_device;
+	ctrl->device->devt = MKDEV(nvme_char_major, ctrl->instance);
+	ctrl->device->class = nvme_class;
+	ctrl->device->parent = ctrl->dev;
+	ctrl->device->groups = nvme_dev_attr_groups;
+	ctrl->device->release = nvme_free_ctrl;
+	dev_set_drvdata(ctrl->device, ctrl);
+	ret = dev_set_name(ctrl->device, "nvme%d", ctrl->instance);
+	if (ret)
 		goto out_release_instance;
-	}
-	get_device(ctrl->device);
+	ret = device_add(ctrl->device);
+	if (ret)
+		goto out_free_name;
+
 	ida_init(&ctrl->ns_ida);
 
 	spin_lock(&dev_list_lock);
@@ -2779,6 +2778,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 		min(default_ps_max_latency_us, (unsigned long)S32_MAX));
 
 	return 0;
+out_free_name:
+	kfree_const(dev->kobj.name);
 out_release_instance:
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 out:
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index c6c903f1b172..aa9aec6923bb 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2692,14 +2692,10 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
 	int ret;
 
-	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
-		return -EBUSY;
-
+	nvme_get_ctrl(&ctrl->ctrl);
 	ret = __nvme_fc_del_ctrl(ctrl);
-
 	if (!ret)
 		flush_workqueue(nvme_wq);
-
 	nvme_put_ctrl(&ctrl->ctrl);
 
 	return ret;
@@ -2918,7 +2914,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 		return ERR_PTR(ret);
 	}
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	dev_info(ctrl->ctrl.device,
 		"NVME-FC{%d}: new ctrl: NQN \"%s\"\n",
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index cb9d93048f3d..ae60d8342e60 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -127,12 +127,12 @@ struct nvme_ctrl {
 	struct request_queue *admin_q;
 	struct request_queue *connect_q;
 	struct device *dev;
-	struct kref kref;
 	int instance;
 	struct blk_mq_tag_set *tagset;
 	struct blk_mq_tag_set *admin_tagset;
 	struct list_head namespaces;
 	struct mutex namespaces_mutex;
+	struct device ctrl_device;
 	struct device *device;	/* char device */
 	struct list_head node;
 	struct ida ns_ida;
@@ -279,6 +279,16 @@ static inline void nvme_end_request(struct request *req, __le16 status,
 	blk_mq_complete_request(req);
 }
 
+static inline void nvme_get_ctrl(struct nvme_ctrl *ctrl)
+{
+	get_device(ctrl->device);
+}
+
+static inline void nvme_put_ctrl(struct nvme_ctrl *ctrl)
+{
+	put_device(ctrl->device);
+}
+
 void nvme_complete_rq(struct request *req);
 void nvme_cancel_request(struct request *req, void *data, bool reserved);
 bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 11e6fd9d0ba4..7735571ffc9a 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2292,7 +2292,7 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev, int status)
 {
 	dev_warn(dev->ctrl.device, "Removing after probe failure status: %d\n", status);
 
-	kref_get(&dev->ctrl.kref);
+	nvme_get_ctrl(&dev->ctrl);
 	nvme_dev_disable(dev, false);
 	if (!schedule_work(&dev->remove_work))
 		nvme_put_ctrl(&dev->ctrl);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 4552744eff45..62b58f9b7d00 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1793,8 +1793,7 @@ static int nvme_rdma_del_ctrl(struct nvme_ctrl *nctrl)
 	 * Keep a reference until all work is flushed since
 	 * __nvme_rdma_del_ctrl can free the ctrl mem
 	 */
-	if (!kref_get_unless_zero(&ctrl->ctrl.kref))
-		return -EBUSY;
+	nvme_get_ctrl(&ctrl->ctrl);
 	ret = __nvme_rdma_del_ctrl(ctrl);
 	if (!ret)
 		flush_work(&ctrl->delete_work);
@@ -1955,7 +1954,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	dev_info(ctrl->ctrl.device, "new ctrl: NQN \"%s\", addr %pISpcs\n",
 		ctrl->ctrl.opts->subsysnqn, &ctrl->addr);
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	mutex_lock(&nvme_rdma_ctrl_mutex);
 	list_add_tail(&ctrl->list, &nvme_rdma_ctrl_list);
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c56354e1e4c6..f83e925fe64a 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -642,7 +642,7 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 	dev_info(ctrl->ctrl.device,
 		 "new ctrl: \"%s\"\n", ctrl->ctrl.opts->subsysnqn);
 
-	kref_get(&ctrl->ctrl.kref);
+	nvme_get_ctrl(&ctrl->ctrl);
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
 	WARN_ON_ONCE(!changed);
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:25 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:25 +0200
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-17-hch@lst.de>

This patch adds native multipath support to the nvme driver.  For each
namespace we create only single block device node, which can be used
to access that namespace through any of the controllers that refer to it.
The gendisk for each controllers path to the name space still exists
inside the kernel, but is hidden from userspace.  The character device
nodes are still available on a per-controller basis.  A new link from
the sysfs directory for the subsystem allows to find all controllers
for a given subsystem.

Currently we will always send I/O to the first available path, this will
be changed once the NVMe Asynchronous Namespace Access (ANA) TP is
ratified and implemented, at which point we will look at the ANA state
for each namespace.  Another possibility that was prototyped is to
use the path that is closes to the submitting NUMA code, which will be
mostly interesting for PCI, but might also be useful for RDMA or FC
transports in the future.  There is not plan to implement round robin
or I/O service time path selectors, as those are not scalable with
the performance rates provided by NVMe.

The multipath device will go away once all paths to it disappear,
any delay to keep it alive needs to be implemented at the controller
level.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c | 398 ++++++++++++++++++++++++++++++++++++++++-------
 drivers/nvme/host/nvme.h |  15 +-
 drivers/nvme/host/pci.c  |   2 +
 3 files changed, 355 insertions(+), 60 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1db26729bd89..22c06cd3bef0 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -102,6 +102,20 @@ static int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
 	return ret;
 }
 
+static void nvme_failover_req(struct request *req)
+{
+	struct nvme_ns *ns = req->q->queuedata;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ns->head->requeue_lock, flags);
+	blk_steal_bios(&ns->head->requeue_list, req);
+	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
+	blk_mq_end_request(req, 0);
+
+	nvme_reset_ctrl(ns->ctrl);
+	kblockd_schedule_work(&ns->head->requeue_work);
+}
+
 static blk_status_t nvme_error_status(struct request *req)
 {
 	switch (nvme_req(req)->status & 0x7ff) {
@@ -129,6 +143,53 @@ static blk_status_t nvme_error_status(struct request *req)
 	}
 }
 
+static bool nvme_req_needs_failover(struct request *req)
+{
+	if (!(req->cmd_flags & REQ_NVME_MPATH))
+		return false;
+
+	switch (nvme_req(req)->status & 0x7ff) {
+	/*
+	 * Generic command status:
+	 */
+	case NVME_SC_INVALID_OPCODE:
+	case NVME_SC_INVALID_FIELD:
+	case NVME_SC_INVALID_NS:
+	case NVME_SC_LBA_RANGE:
+	case NVME_SC_CAP_EXCEEDED:
+	case NVME_SC_RESERVATION_CONFLICT:
+		return false;
+
+	/*
+	 * I/O command set specific error.  Unfortunately these values are
+	 * reused for fabrics commands, but those should never get here.
+	 */
+	case NVME_SC_BAD_ATTRIBUTES:
+	case NVME_SC_INVALID_PI:
+	case NVME_SC_READ_ONLY:
+	case NVME_SC_ONCS_NOT_SUPPORTED:
+		WARN_ON_ONCE(nvme_req(req)->cmd->common.opcode ==
+			nvme_fabrics_command);
+		return false;
+
+	/*
+	 * Media and Data Integrity Errors:
+	 */
+	case NVME_SC_WRITE_FAULT:
+	case NVME_SC_READ_ERROR:
+	case NVME_SC_GUARD_CHECK:
+	case NVME_SC_APPTAG_CHECK:
+	case NVME_SC_REFTAG_CHECK:
+	case NVME_SC_COMPARE_FAILED:
+	case NVME_SC_ACCESS_DENIED:
+	case NVME_SC_UNWRITTEN_BLOCK:
+		return false;
+	}
+
+	/* Everything else could be a path failure, so should be retried */
+	return true;
+}
+
 static inline bool nvme_req_needs_retry(struct request *req)
 {
 	if (blk_noretry_request(req))
@@ -143,6 +204,11 @@ static inline bool nvme_req_needs_retry(struct request *req)
 void nvme_complete_rq(struct request *req)
 {
 	if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
+		if (nvme_req_needs_failover(req)) {
+			nvme_failover_req(req);
+			return;
+		}
+
 		nvme_req(req)->retries++;
 		blk_mq_requeue_request(req, true);
 		return;
@@ -171,6 +237,18 @@ void nvme_cancel_request(struct request *req, void *data, bool reserved)
 }
 EXPORT_SYMBOL_GPL(nvme_cancel_request);
 
+static void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+
+	mutex_lock(&ctrl->namespaces_mutex);
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (ns->head)
+			kblockd_schedule_work(&ns->head->requeue_work);
+	}
+	mutex_unlock(&ctrl->namespaces_mutex);
+}
+
 bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 		enum nvme_ctrl_state new_state)
 {
@@ -238,9 +316,10 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 
 	if (changed)
 		ctrl->state = new_state;
-
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 
+	if (changed && ctrl->state == NVME_CTRL_LIVE)
+		nvme_kick_requeue_lists(ctrl);
 	return changed;
 }
 EXPORT_SYMBOL_GPL(nvme_change_ctrl_state);
@@ -250,6 +329,15 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	del_gendisk(head->disk);
+	blk_set_queue_dying(head->disk->queue);
+	/* make sure all pending bios are cleaned up */
+	kblockd_schedule_work(&head->requeue_work);
+	flush_work(&head->requeue_work);
+	blk_cleanup_queue(head->disk->queue);
+	put_disk(head->disk);
+	ida_simple_remove(&head->subsys->ns_ida, head->instance);
+
 	list_del_init(&head->entry);
 	cleanup_srcu_struct(&head->srcu);
 	kfree(head);
@@ -266,9 +354,7 @@ static void nvme_free_ns(struct kref *kref)
 
 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
-
 	put_disk(ns->disk);
-	ida_simple_remove(&ns->ctrl->ns_ida, ns->instance);
 	nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
 	kfree(ns);
@@ -1013,11 +1099,9 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	return status;
 }
 
-static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
-		unsigned int cmd, unsigned long arg)
+static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd,
+		unsigned long arg)
 {
-	struct nvme_ns *ns = bdev->bd_disk->private_data;
-
 	switch (cmd) {
 	case NVME_IOCTL_ID:
 		force_successful_syscall_return();
@@ -1040,18 +1124,10 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 	}
 }
 
+/* should never be called due to GENHD_FL_HIDDEN */
 static int nvme_open(struct block_device *bdev, fmode_t mode)
 {
-	struct nvme_ns *ns = bdev->bd_disk->private_data;
-
-	if (!kref_get_unless_zero(&ns->kref))
-		return -ENXIO;
-	return 0;
-}
-
-static void nvme_release(struct gendisk *disk, fmode_t mode)
-{
-	nvme_put_ns(disk->private_data);
+	return WARN_ON_ONCE(-ENXIO);
 }
 
 static int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo)
@@ -1081,8 +1157,10 @@ static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
 	if (blk_get_integrity(disk) &&
 	    (ns->pi_type != pi_type || ns->ms != old_ms ||
 	     bs != queue_logical_block_size(disk->queue) ||
-	     (ns->ms && ns->ext)))
+	     (ns->ms && ns->ext))) {
 		blk_integrity_unregister(disk);
+		blk_integrity_unregister(ns->head->disk);
+	}
 
 	ns->pi_type = pi_type;
 }
@@ -1110,7 +1188,9 @@ static void nvme_init_integrity(struct nvme_ns *ns)
 	}
 	integrity.tuple_size = ns->ms;
 	blk_integrity_register(ns->disk, &integrity);
+	blk_integrity_register(ns->head->disk, &integrity);
 	blk_queue_max_integrity_segments(ns->queue, 1);
+	blk_queue_max_integrity_segments(ns->head->disk->queue, 1);
 }
 #else
 static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
@@ -1128,7 +1208,7 @@ static void nvme_set_chunk_size(struct nvme_ns *ns)
 	blk_queue_chunk_sectors(ns->queue, rounddown_pow_of_two(chunk_size));
 }
 
-static void nvme_config_discard(struct nvme_ns *ns)
+static void nvme_config_discard(struct nvme_ns *ns, struct request_queue *queue)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	u32 logical_block_size = queue_logical_block_size(ns->queue);
@@ -1139,18 +1219,18 @@ static void nvme_config_discard(struct nvme_ns *ns)
 	if (ctrl->nr_streams && ns->sws && ns->sgs) {
 		unsigned int sz = logical_block_size * ns->sws * ns->sgs;
 
-		ns->queue->limits.discard_alignment = sz;
-		ns->queue->limits.discard_granularity = sz;
+		queue->limits.discard_alignment = sz;
+		queue->limits.discard_granularity = sz;
 	} else {
 		ns->queue->limits.discard_alignment = logical_block_size;
 		ns->queue->limits.discard_granularity = logical_block_size;
 	}
-	blk_queue_max_discard_sectors(ns->queue, UINT_MAX);
-	blk_queue_max_discard_segments(ns->queue, NVME_DSM_MAX_RANGES);
-	queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, ns->queue);
+	blk_queue_max_discard_sectors(queue, UINT_MAX);
+	blk_queue_max_discard_segments(queue, NVME_DSM_MAX_RANGES);
+	queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, queue);
 
 	if (ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES)
-		blk_queue_max_write_zeroes_sectors(ns->queue, UINT_MAX);
+		blk_queue_max_write_zeroes_sectors(queue, UINT_MAX);
 }
 
 static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
@@ -1207,17 +1287,25 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
 	if (ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)
 		nvme_prep_integrity(disk, id, bs);
 	blk_queue_logical_block_size(ns->queue, bs);
+	blk_queue_logical_block_size(ns->head->disk->queue, bs);
 	if (ns->noiob)
 		nvme_set_chunk_size(ns);
 	if (ns->ms && !blk_get_integrity(disk) && !ns->ext)
 		nvme_init_integrity(ns);
-	if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk))
+	if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk)) {
 		set_capacity(disk, 0);
-	else
+		if (ns->head)
+			set_capacity(ns->head->disk, 0);
+	} else {
 		set_capacity(disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
+		if (ns->head)
+			set_capacity(ns->head->disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9));
+	}
 
-	if (ctrl->oncs & NVME_CTRL_ONCS_DSM)
-		nvme_config_discard(ns);
+	if (ctrl->oncs & NVME_CTRL_ONCS_DSM) {
+		nvme_config_discard(ns, ns->queue);
+		nvme_config_discard(ns, ns->head->disk->queue);
+	}
 	blk_mq_unfreeze_queue(disk->queue);
 }
 
@@ -1255,6 +1343,29 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 	return ret;
 }
 
+static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head)
+{
+	struct nvme_ns *ns;
+
+	list_for_each_entry_rcu(ns, &head->list, siblings) {
+		if (ns->ctrl->state == NVME_CTRL_LIVE) {
+			rcu_assign_pointer(head->current_path, ns);
+			return ns;
+		}
+	}
+
+	return NULL;
+}
+
+static inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
+{
+	struct nvme_ns *ns = srcu_dereference(head->current_path, &head->srcu);
+
+	if (unlikely(!ns || ns->ctrl->state != NVME_CTRL_LIVE))
+		ns = __nvme_find_path(head);
+	return ns;
+}
+
 static char nvme_pr_type(enum pr_type type)
 {
 	switch (type) {
@@ -1278,8 +1389,10 @@ static char nvme_pr_type(enum pr_type type)
 static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 				u64 key, u64 sa_key, u8 op)
 {
-	struct nvme_ns *ns = bdev->bd_disk->private_data;
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+	struct nvme_ns *ns;
 	struct nvme_command c;
+	int srcu_idx, ret;
 	u8 data[16] = { 0, };
 
 	put_unaligned_le64(key, &data[0]);
@@ -1287,10 +1400,17 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = op;
-	c.common.nsid = cpu_to_le32(ns->head->ns_id);
+	c.common.nsid = cpu_to_le32(head->ns_id);
 	c.common.cdw10[0] = cpu_to_le32(cdw10);
 
-	return nvme_submit_sync_cmd(ns->queue, &c, data, 16);
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns))
+		ret = nvme_submit_sync_cmd(ns->queue, &c, data, 16);
+	else
+		ret = -EWOULDBLOCK;
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
 }
 
 static int nvme_pr_register(struct block_device *bdev, u64 old,
@@ -1369,15 +1489,16 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 EXPORT_SYMBOL_GPL(nvme_sec_submit);
 #endif /* CONFIG_BLK_SED_OPAL */
 
+/*
+ * While we don't expose the per-controller devices to userspace we still
+ * need valid file operations for them, for one because the block layer
+ * expects to use the owner field for module refcounting, and also because
+ * we call revalidate_disk internally.
+ */
 static const struct block_device_operations nvme_fops = {
 	.owner		= THIS_MODULE,
-	.ioctl		= nvme_ioctl,
-	.compat_ioctl	= nvme_ioctl,
 	.open		= nvme_open,
-	.release	= nvme_release,
-	.getgeo		= nvme_getgeo,
 	.revalidate_disk= nvme_revalidate_disk,
-	.pr_ops		= &nvme_pr_ops,
 };
 
 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u64 cap, bool enabled)
@@ -1774,6 +1895,7 @@ static void nvme_destroy_subsystem(struct kref *ref)
 	list_del(&subsys->entry);
 	mutex_unlock(&nvme_subsystems_lock);
 
+	ida_destroy(&subsys->ns_ida);
 	device_del(&subsys->dev);
 	put_device(&subsys->dev);
 }
@@ -1803,7 +1925,7 @@ static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
 static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 {
 	struct nvme_subsystem *subsys, *found;
-	int ret;
+	int ret = -ENOMEM;
 
 	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
 	if (!subsys)
@@ -1854,12 +1976,21 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 				"failed to register subsystem device.\n");
 			goto out_unlock;
 		}
+		ida_init(&subsys->ns_ida);
 		list_add_tail(&subsys->entry, &nvme_subsystems);
 	}
 
 	ctrl->subsys = subsys;
 	mutex_unlock(&nvme_subsystems_lock);
 
+	if (sysfs_create_link(&subsys->dev.kobj, &ctrl->device->kobj,
+			dev_name(ctrl->device))) {
+		dev_err(ctrl->device,
+			"failed to create sysfs link from subsystem.\n");
+		/* the transport driver will eventually put the subsystem */
+		return -EINVAL;
+	}
+
 	mutex_lock(&subsys->lock);
 	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
 	mutex_unlock(&subsys->lock);
@@ -2375,6 +2506,121 @@ static const struct attribute_group *nvme_dev_attr_groups[] = {
 	NULL,
 };
 
+static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,
+		struct bio *bio)
+{
+	struct nvme_ns_head *head = q->queuedata;
+	struct device *dev = disk_to_dev(head->disk);
+	struct nvme_ns *ns;
+	blk_qc_t ret = BLK_QC_T_NONE;
+	int srcu_idx;
+
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns)) {
+		bio->bi_disk = ns->disk;
+		bio->bi_opf |= REQ_NVME_MPATH;
+		ret = direct_make_request(bio);
+	} else if (!list_empty_careful(&head->list)) {
+		dev_warn_ratelimited(dev, "no path available - requeing I/O\n");
+
+		spin_lock_irq(&head->requeue_lock);
+		bio_list_add(&head->requeue_list, bio);
+		spin_unlock_irq(&head->requeue_lock);
+	} else {
+		dev_warn_ratelimited(dev, "no path - failing I/O\n");
+
+		bio->bi_status = BLK_STS_IOERR;
+		bio_endio(bio);
+	}
+
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
+}
+
+static bool nvme_ns_head_poll(struct request_queue *q, blk_qc_t qc)
+{
+	struct nvme_ns_head *head = q->queuedata;
+	struct nvme_ns *ns;
+	bool found = false;
+	int srcu_idx;
+
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = srcu_dereference(head->current_path, &head->srcu);
+	if (likely(ns && ns->ctrl->state == NVME_CTRL_LIVE))
+		found = ns->queue->poll_fn(q, qc);
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return found;
+}
+
+static int nvme_ns_head_open(struct block_device *bdev, fmode_t mode)
+{
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+
+	if (!kref_get_unless_zero(&head->ref))
+		return -ENXIO;
+	return 0;
+}
+
+static void nvme_ns_head_release(struct gendisk *disk, fmode_t mode)
+{
+	nvme_put_ns_head(disk->private_data);
+}
+
+/*
+ * Issue the ioctl on the first available path.  Note that unlike normal block
+ * layer requests we will not retry failed request on another controller.
+ */
+static int nvme_ns_head_ioctl(struct block_device *bdev, fmode_t mode,
+		unsigned int cmd, unsigned long arg)
+{
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+	struct nvme_ns *ns;
+	int srcu_idx, ret;
+
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head);
+	if (likely(ns))
+		ret = nvme_ns_ioctl(ns, cmd, arg);
+	else
+		ret = -EWOULDBLOCK;
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
+}
+
+static const struct block_device_operations nvme_ns_head_ops = {
+	.owner		= THIS_MODULE,
+	.open		= nvme_ns_head_open,
+	.release	= nvme_ns_head_release,
+	.ioctl		= nvme_ns_head_ioctl,
+	.compat_ioctl	= nvme_ns_head_ioctl,
+	.getgeo		= nvme_getgeo,
+	.pr_ops		= &nvme_pr_ops,
+};
+
+static void nvme_requeue_work(struct work_struct *work)
+{
+	struct nvme_ns_head *head =
+		container_of(work, struct nvme_ns_head, requeue_work);
+	struct bio *bio, *next;
+
+	spin_lock_irq(&head->requeue_lock);
+	next = bio_list_get(&head->requeue_list);
+	spin_unlock_irq(&head->requeue_lock);
+
+	while ((bio = next) != NULL) {
+		next = bio->bi_next;
+		bio->bi_next = NULL;
+
+		/*
+		 * Reset disk to the mpath node and resubmit to select a new
+		 * path.
+		 */
+		bio->bi_disk = head->disk;
+		generic_make_request(bio);
+	}
+}
+
 static struct nvme_ns_head *__nvme_find_ns_head(struct nvme_subsystem *subsys,
 		unsigned nsid)
 {
@@ -2410,15 +2656,23 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 		unsigned nsid, struct nvme_id_ns *id)
 {
 	struct nvme_ns_head *head;
+	struct request_queue *q;
 	int ret = -ENOMEM;
 
 	head = kzalloc(sizeof(*head), GFP_KERNEL);
 	if (!head)
 		goto out;
-
+	ret = ida_simple_get(&ctrl->subsys->ns_ida, 1, 0, GFP_KERNEL);
+	if (ret < 0)
+		goto out_free_head;
+	head->instance = ret;
 	INIT_LIST_HEAD(&head->list);
 	head->ns_id = nsid;
+	bio_list_init(&head->requeue_list);
+	spin_lock_init(&head->requeue_lock);
+	INIT_WORK(&head->requeue_work, nvme_requeue_work);
 	init_srcu_struct(&head->srcu);
+	head->subsys = ctrl->subsys;
 	kref_init(&head->ref);
 
 	nvme_report_ns_ids(ctrl, nsid, id, &head->ids);
@@ -2427,20 +2681,46 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 	if (ret) {
 		dev_err(ctrl->device,
 			"duplicate IDs for nsid %d\n", nsid);
-		goto out_free_head;
+		goto out_release_instance;
 	}
 
+	ret = -ENOMEM;
+	q = blk_alloc_queue_node(GFP_KERNEL, NUMA_NO_NODE);
+	if (!q)
+		goto out_free_head;
+	q->queuedata = head;
+	blk_queue_make_request(q, nvme_ns_head_make_request);
+	q->poll_fn = nvme_ns_head_poll;
+	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
+	/* set to a default value for 512 until disk is validated */
+	blk_queue_logical_block_size(q, 512);
+	nvme_set_queue_limits(ctrl, q);
+
+	head->disk = alloc_disk(0);
+	if (!head->disk)
+		goto out_cleanup_queue;
+	head->disk->fops = &nvme_ns_head_ops;
+	head->disk->private_data = head;
+	head->disk->queue = q;
+	head->disk->flags = GENHD_FL_EXT_DEVT;
+	sprintf(head->disk->disk_name, "nvme%dn%d",
+			ctrl->subsys->instance, nsid);
 	list_add_tail(&head->entry, &ctrl->subsys->nsheads);
 	return head;
+
+out_cleanup_queue:
+	blk_cleanup_queue(q);
 out_free_head:
 	cleanup_srcu_struct(&head->srcu);
 	kfree(head);
+out_release_instance:
+	ida_simple_remove(&ctrl->subsys->ns_ida, head->instance);
 out:
 	return ERR_PTR(ret);
 }
 
 static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
-		struct nvme_id_ns *id)
+		struct nvme_id_ns *id, bool *new)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	bool is_shared = id->nmic & (1 << 0);
@@ -2456,6 +2736,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 			ret = PTR_ERR(head);
 			goto out_unlock;
 		}
+
+		*new = true;
 	} else {
 		struct nvme_ns_ids ids;
 
@@ -2467,6 +2749,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 			ret = -EINVAL;
 			goto out_unlock;
 		}
+
+		*new = false;
 	}
 
 	list_add_tail(&ns->siblings, &head->list);
@@ -2537,18 +2821,15 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	struct nvme_id_ns *id;
 	char disk_name[DISK_NAME_LEN];
 	int node = dev_to_node(ctrl->dev);
+	bool new = true;
 
 	ns = kzalloc_node(sizeof(*ns), GFP_KERNEL, node);
 	if (!ns)
 		return;
 
-	ns->instance = ida_simple_get(&ctrl->ns_ida, 1, 0, GFP_KERNEL);
-	if (ns->instance < 0)
-		goto out_free_ns;
-
 	ns->queue = blk_mq_init_queue(ctrl->tagset);
 	if (IS_ERR(ns->queue))
-		goto out_release_instance;
+		goto out_free_ns;
 	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, ns->queue);
 	ns->queue->queuedata = ns;
 	ns->ctrl = ctrl;
@@ -2560,8 +2841,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	nvme_set_queue_limits(ctrl, ns->queue);
 	nvme_setup_streams_ns(ctrl, ns);
 
-	sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->instance);
-
 	id = nvme_identify_ns(ctrl, nsid);
 	if (!id)
 		goto out_free_queue;
@@ -2569,9 +2848,11 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	if (nvme_init_ns_head(ns, nsid, id))
+	if (nvme_init_ns_head(ns, nsid, id, &new))
 		goto out_free_id;
 
+	sprintf(disk_name, "nvme%dc%dn%d", ctrl->subsys->instance,
+			ctrl->cntlid, ns->head->instance);
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
 			dev_warn(ctrl->device, "LightNVM init failure\n");
@@ -2586,7 +2867,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	disk->fops = &nvme_fops;
 	disk->private_data = ns;
 	disk->queue = ns->queue;
-	disk->flags = GENHD_FL_EXT_DEVT;
+	disk->flags = GENHD_FL_HIDDEN;
 	memcpy(disk->disk_name, disk_name, DISK_NAME_LEN);
 	ns->disk = disk;
 
@@ -2608,6 +2889,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (ns->ndev && nvme_nvm_register_sysfs(ns))
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
+
+	if (new)
+		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
+
 	return;
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
@@ -2617,8 +2902,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	kfree(id);
  out_free_queue:
 	blk_cleanup_queue(ns->queue);
- out_release_instance:
-	ida_simple_remove(&ctrl->ns_ida, ns->instance);
  out_free_ns:
 	kfree(ns);
 }
@@ -2633,8 +2916,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		if (blk_get_integrity(ns->disk))
 			blk_integrity_unregister(ns->disk);
-		sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
-					&nvme_ns_attr_group);
 		if (ns->ndev)
 			nvme_nvm_unregister_sysfs(ns);
 		del_gendisk(ns->disk);
@@ -2642,8 +2923,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	}
 
 	mutex_lock(&ns->ctrl->subsys->lock);
-	if (head)
+	if (head) {
+		if (head->current_path == ns)
+			rcu_assign_pointer(head->current_path, NULL);
 		list_del_rcu(&ns->siblings);
+	}
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	mutex_lock(&ns->ctrl->namespaces_mutex);
@@ -2948,12 +3232,12 @@ static void nvme_free_ctrl(struct device *dev)
 	struct nvme_subsystem *subsys = ctrl->subsys;
 
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
-	ida_destroy(&ctrl->ns_ida);
 
 	if (subsys) {
 		mutex_lock(&subsys->lock);
 		list_del(&ctrl->subsys_entry);
 		mutex_unlock(&subsys->lock);
+		sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
 	}
 
 	ctrl->ops->free_ctrl(ctrl);
@@ -3006,8 +3290,6 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	if (ret)
 		goto out_free_name;
 
-	ida_init(&ctrl->ns_ida);
-
 	/*
 	 * Initialize latency tolerance controls.  The sysfs files won't
 	 * be visible to userspace unless the device actually supports APST.
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 849413def126..87dd77e8fbdf 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -95,6 +95,11 @@ struct nvme_request {
 	u16			status;
 };
 
+/*
+ * Mark a bio as coming in through the mpath node.
+ */
+#define REQ_NVME_MPATH		REQ_DRV
+
 enum {
 	NVME_REQ_CANCELLED		= (1 << 0),
 };
@@ -136,7 +141,6 @@ struct nvme_ctrl {
 	struct device ctrl_device;
 	struct device *device;	/* char device */
 	struct cdev cdev;
-	struct ida ns_ida;
 	struct work_struct reset_work;
 
 	struct nvme_subsystem *subsys;
@@ -213,6 +217,7 @@ struct nvme_subsystem {
 	char			model[40];
 	char			firmware_rev[8];
 	u16			vendor_id;
+	struct ida		ns_ida;
 };
 
 /*
@@ -232,12 +237,19 @@ struct nvme_ns_ids {
  * only ever has a single entry for private namespaces.
  */
 struct nvme_ns_head {
+	struct nvme_ns __rcu	*current_path;
+	struct gendisk		*disk;
 	struct list_head	list;
 	struct srcu_struct      srcu;
+	struct nvme_subsystem	*subsys;
+	struct bio_list		requeue_list;
+	spinlock_t		requeue_lock;
+	struct work_struct	requeue_work;
 	unsigned		ns_id;
 	struct nvme_ns_ids	ids;
 	struct list_head	entry;
 	struct kref		ref;
+	int			instance;
 };
 
 struct nvme_ns {
@@ -250,7 +262,6 @@ struct nvme_ns {
 	struct nvm_dev *ndev;
 	struct kref kref;
 	struct nvme_ns_head *head;
-	int instance;
 
 	int lba_shift;
 	u16 ms;
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 7735571ffc9a..bbece5edabff 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1050,6 +1050,8 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
 {
 	struct nvme_queue *nvmeq = hctx->driver_data;
 
+	printk_ratelimited("%s: called\n", __func__);
+
 	return __nvme_poll(nvmeq, tag);
 }
 
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:20 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:20 +0200
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-12-hch@lst.de>

Use the core chrdev code to set up the link between the character device
and the nvme controller.  This allows us to get rid of the global list
of all controllers, and also ensures that we have both a reference to
the controller and the transport module before the open method of the
character device is called.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sgi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 76 ++++++++++--------------------------------------
 drivers/nvme/host/nvme.h |  3 +-
 2 files changed, 18 insertions(+), 61 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3a97daa163f6..a56a1e0432e7 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -52,9 +52,6 @@ static u8 nvme_max_retries = 5;
 module_param_named(max_retries, nvme_max_retries, byte, 0644);
 MODULE_PARM_DESC(max_retries, "max number of retries a command may have");
 
-static int nvme_char_major;
-module_param(nvme_char_major, int, 0);
-
 static unsigned long default_ps_max_latency_us = 100000;
 module_param(default_ps_max_latency_us, ulong, 0644);
 MODULE_PARM_DESC(default_ps_max_latency_us,
@@ -71,11 +68,8 @@ MODULE_PARM_DESC(streams, "turn on support for Streams write directives");
 struct workqueue_struct *nvme_wq;
 EXPORT_SYMBOL_GPL(nvme_wq);
 
-static LIST_HEAD(nvme_ctrl_list);
-static DEFINE_SPINLOCK(dev_list_lock);
-
 static DEFINE_IDA(nvme_instance_ida);
-
+static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
@@ -1031,20 +1025,12 @@ static int nvme_open(struct block_device *bdev, fmode_t mode)
 
 	if (!kref_get_unless_zero(&ns->kref))
 		return -ENXIO;
-	if (!try_module_get(ns->ctrl->ops->module)) {
-		kref_put(&ns->kref, nvme_free_ns);
-		return -ENXIO;
-	}
-
 	return 0;
 }
 
 static void nvme_release(struct gendisk *disk, fmode_t mode)
 {
-	struct nvme_ns *ns = disk->private_data;
-
-	module_put(ns->ctrl->ops->module);
-	nvme_put_ns(ns);
+	nvme_put_ns(disk->private_data);
 }
 
 static int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo)
@@ -1902,33 +1888,12 @@ EXPORT_SYMBOL_GPL(nvme_init_identify);
 
 static int nvme_dev_open(struct inode *inode, struct file *file)
 {
-	struct nvme_ctrl *ctrl;
-	int instance = iminor(inode);
-	int ret = -ENODEV;
-
-	spin_lock(&dev_list_lock);
-	list_for_each_entry(ctrl, &nvme_ctrl_list, node) {
-		if (ctrl->instance != instance)
-			continue;
-
-		if (!ctrl->admin_q) {
-			ret = -EWOULDBLOCK;
-			break;
-		}
-		if (!kobject_get_unless_zero(&ctrl->device->kobj))
-			break;
-		file->private_data = ctrl;
-		ret = 0;
-		break;
-	}
-	spin_unlock(&dev_list_lock);
-
-	return ret;
-}
+	struct nvme_ctrl *ctrl =
+		container_of(inode->i_cdev, struct nvme_ctrl, cdev);
 
-static int nvme_dev_release(struct inode *inode, struct file *file)
-{
-	nvme_put_ctrl(file->private_data);
+	if (!ctrl->admin_q)
+		return -EWOULDBLOCK;
+	file->private_data = ctrl;
 	return 0;
 }
 
@@ -1992,7 +1957,6 @@ static long nvme_dev_ioctl(struct file *file, unsigned int cmd,
 static const struct file_operations nvme_dev_fops = {
 	.owner		= THIS_MODULE,
 	.open		= nvme_dev_open,
-	.release	= nvme_dev_release,
 	.unlocked_ioctl	= nvme_dev_ioctl,
 	.compat_ioctl	= nvme_dev_ioctl,
 };
@@ -2703,11 +2667,7 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
-	device_del(ctrl->device);
-
-	spin_lock(&dev_list_lock);
-	list_del(&ctrl->node);
-	spin_unlock(&dev_list_lock);
+	cdev_device_del(&ctrl->cdev, ctrl->device);
 }
 EXPORT_SYMBOL_GPL(nvme_uninit_ctrl);
 
@@ -2750,7 +2710,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	device_initialize(&ctrl->ctrl_device);
 	ctrl->device = &ctrl->ctrl_device;
-	ctrl->device->devt = MKDEV(nvme_char_major, ctrl->instance);
+	ctrl->device->devt = MKDEV(MAJOR(nvme_chr_devt), ctrl->instance);
 	ctrl->device->class = nvme_class;
 	ctrl->device->parent = ctrl->dev;
 	ctrl->device->groups = nvme_dev_attr_groups;
@@ -2759,16 +2719,15 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	ret = dev_set_name(ctrl->device, "nvme%d", ctrl->instance);
 	if (ret)
 		goto out_release_instance;
-	ret = device_add(ctrl->device);
+
+	cdev_init(&ctrl->cdev, &nvme_dev_fops);
+	ctrl->cdev.owner = ops->module;
+	ret = cdev_device_add(&ctrl->cdev, ctrl->device);
 	if (ret)
 		goto out_free_name;
 
 	ida_init(&ctrl->ns_ida);
 
-	spin_lock(&dev_list_lock);
-	list_add_tail(&ctrl->node, &nvme_ctrl_list);
-	spin_unlock(&dev_list_lock);
-
 	/*
 	 * Initialize latency tolerance controls.  The sysfs files won't
 	 * be visible to userspace unless the device actually supports APST.
@@ -2909,12 +2868,9 @@ int __init nvme_core_init(void)
 	if (!nvme_wq)
 		return -ENOMEM;
 
-	result = __register_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme",
-							&nvme_dev_fops);
+	result = alloc_chrdev_region(&nvme_chr_devt, 0, NVME_MINORS, "nvme");
 	if (result < 0)
 		goto destroy_wq;
-	else if (result > 0)
-		nvme_char_major = result;
 
 	nvme_class = class_create(THIS_MODULE, "nvme");
 	if (IS_ERR(nvme_class)) {
@@ -2925,7 +2881,7 @@ int __init nvme_core_init(void)
 	return 0;
 
 unregister_chrdev:
-	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 destroy_wq:
 	destroy_workqueue(nvme_wq);
 	return result;
@@ -2934,7 +2890,7 @@ int __init nvme_core_init(void)
 void nvme_core_exit(void)
 {
 	class_destroy(nvme_class);
-	__unregister_chrdev(nvme_char_major, 0, NVME_MINORS, "nvme");
+	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 	destroy_workqueue(nvme_wq);
 }
 
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index ae60d8342e60..1bb2bc165e54 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -15,6 +15,7 @@
 #define _NVME_H
 
 #include <linux/nvme.h>
+#include <linux/cdev.h>
 #include <linux/pci.h>
 #include <linux/kref.h>
 #include <linux/blk-mq.h>
@@ -134,7 +135,7 @@ struct nvme_ctrl {
 	struct mutex namespaces_mutex;
 	struct device ctrl_device;
 	struct device *device;	/* char device */
-	struct list_head node;
+	struct cdev cdev;
 	struct ida ns_ida;
 	struct work_struct reset_work;
 
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:21 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:21 +0200
Subject: [PATCH 12/17] nvme: check for a live controller in nvme_dev_open
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-13-hch@lst.de>

This is a much more sensible check than just the admin queue.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at rimbeg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index a56a1e0432e7..df525ab42fcd 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1891,7 +1891,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file)
 	struct nvme_ctrl *ctrl =
 		container_of(inode->i_cdev, struct nvme_ctrl, cdev);
 
-	if (!ctrl->admin_q)
+	if (ctrl->state != NVME_CTRL_LIVE)
 		return -EWOULDBLOCK;
 	file->private_data = ctrl;
 	return 0;
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:22 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:22 +0200
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-14-hch@lst.de>

This adds a new nvme_subsystem structure so that we can track multiple
controllers that belong to a single subsystem.  For now we only use it
to store the NQN, and to check that we don't have duplicate NQNs unless
the involved subsystems support multiple controllers.

Includes code originally from Hannes Reinecke to expose the subsystems
in sysfs.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c    | 200 +++++++++++++++++++++++++++++++++++++-------
 drivers/nvme/host/fabrics.c |   4 +-
 drivers/nvme/host/nvme.h    |  26 ++++--
 3 files changed, 194 insertions(+), 36 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index df525ab42fcd..b3d468c77684 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -68,9 +68,14 @@ MODULE_PARM_DESC(streams, "turn on support for Streams write directives");
 struct workqueue_struct *nvme_wq;
 EXPORT_SYMBOL_GPL(nvme_wq);
 
+static DEFINE_IDA(nvme_subsystems_ida);
+static LIST_HEAD(nvme_subsystems);
+static DEFINE_MUTEX(nvme_subsystems_lock);
+
 static DEFINE_IDA(nvme_instance_ida);
 static dev_t nvme_chr_devt;
 static struct class *nvme_class;
+static struct class *nvme_subsys_class;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
 {
@@ -1694,14 +1699,15 @@ static bool quirk_matches(const struct nvme_id_ctrl *id,
 		string_matches(id->fr, q->fr, sizeof(id->fr));
 }
 
-static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+static void nvme_init_subnqn(struct nvme_subsystem *subsys, struct nvme_ctrl *ctrl,
+		struct nvme_id_ctrl *id)
 {
 	size_t nqnlen;
 	int off;
 
 	nqnlen = strnlen(id->subnqn, NVMF_NQN_SIZE);
 	if (nqnlen > 0 && nqnlen < NVMF_NQN_SIZE) {
-		strcpy(ctrl->subnqn, id->subnqn);
+		strcpy(subsys->subnqn, id->subnqn);
 		return;
 	}
 
@@ -1709,14 +1715,131 @@ static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 		dev_warn(ctrl->device, "missing or invalid SUBNQN field.\n");
 
 	/* Generate a "fake" NQN per Figure 254 in NVMe 1.3 + ECN 001 */
-	off = snprintf(ctrl->subnqn, NVMF_NQN_SIZE,
+	off = snprintf(subsys->subnqn, NVMF_NQN_SIZE,
 			"nqn.2014.08.org.nvmexpress:%4x%4x",
 			le16_to_cpu(id->vid), le16_to_cpu(id->ssvid));
-	memcpy(ctrl->subnqn + off, id->sn, sizeof(id->sn));
+	memcpy(subsys->subnqn + off, id->sn, sizeof(id->sn));
 	off += sizeof(id->sn);
-	memcpy(ctrl->subnqn + off, id->mn, sizeof(id->mn));
+	memcpy(subsys->subnqn + off, id->mn, sizeof(id->mn));
 	off += sizeof(id->mn);
-	memset(ctrl->subnqn + off, 0, sizeof(ctrl->subnqn) - off);
+	memset(subsys->subnqn + off, 0, sizeof(subsys->subnqn) - off);
+}
+
+static void __nvme_release_subsystem(struct nvme_subsystem *subsys)
+{
+	ida_simple_remove(&nvme_subsystems_ida, subsys->instance);
+	kfree(subsys);
+}
+
+static void nvme_release_subsystem(struct device *dev)
+{
+	__nvme_release_subsystem(container_of(dev, struct nvme_subsystem, dev));
+}
+
+static void nvme_destroy_subsystem(struct kref *ref)
+{
+	struct nvme_subsystem *subsys =
+			container_of(ref, struct nvme_subsystem, ref);
+
+	mutex_lock(&nvme_subsystems_lock);
+	list_del(&subsys->entry);
+	mutex_unlock(&nvme_subsystems_lock);
+
+	device_del(&subsys->dev);
+	put_device(&subsys->dev);
+}
+
+static void nvme_put_subsystem(struct nvme_subsystem *subsys)
+{
+	kref_put(&subsys->ref, nvme_destroy_subsystem);
+}
+
+static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
+{
+	struct nvme_subsystem *subsys;
+
+	lockdep_assert_held(&nvme_subsystems_lock);
+
+	list_for_each_entry(subsys, &nvme_subsystems, entry) {
+		if (strcmp(subsys->subnqn, subsysnqn))
+			continue;
+		if (!kref_get_unless_zero(&subsys->ref))
+			continue;
+		return subsys;
+	}
+
+	return NULL;
+}
+
+static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
+{
+	struct nvme_subsystem *subsys, *found;
+	int ret;
+
+	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
+	if (!subsys)
+		return -ENOMEM;
+	ret = ida_simple_get(&nvme_subsystems_ida, 0, 0, GFP_KERNEL);
+	if (ret < 0) {
+		kfree(subsys);
+		return ret;
+	}
+	subsys->instance = ret;
+	kref_init(&subsys->ref);
+	INIT_LIST_HEAD(&subsys->ctrls);
+	nvme_init_subnqn(subsys, ctrl, id);
+	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
+	memcpy(subsys->model, id->mn, sizeof(subsys->model));
+	memcpy(subsys->firmware_rev, id->fr, sizeof(subsys->firmware_rev));
+	subsys->vendor_id = le16_to_cpu(id->vid);
+	mutex_init(&subsys->lock);
+
+	subsys->dev.class = nvme_subsys_class;
+	subsys->dev.release = nvme_release_subsystem;
+	dev_set_name(&subsys->dev, "nvme-subsys%d", subsys->instance);
+	device_initialize(&subsys->dev);
+
+	mutex_lock(&nvme_subsystems_lock);
+	found = __nvme_find_get_subsystem(subsys->subnqn);
+	if (found) {
+		/*
+		 * Verify that the subsystem actually supports multiple
+		 * controllers, else bail out.
+		 */
+		if (!(id->cmic & (1 << 1))) {
+			dev_err(ctrl->device,
+				"ignoring ctrl due to duplicate subnqn (%s).\n",
+				found->subnqn);
+			nvme_put_subsystem(found);
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		__nvme_release_subsystem(subsys);
+		subsys = found;
+	} else {
+		ret = device_add(&subsys->dev);
+		if (ret) {
+			dev_err(ctrl->device,
+				"failed to register subsystem device.\n");
+			goto out_unlock;
+		}
+		list_add_tail(&subsys->entry, &nvme_subsystems);
+	}
+
+	ctrl->subsys = subsys;
+	mutex_unlock(&nvme_subsystems_lock);
+
+	mutex_lock(&subsys->lock);
+	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
+	mutex_unlock(&subsys->lock);
+
+	return 0;
+
+out_unlock:
+	mutex_unlock(&nvme_subsystems_lock);
+	put_device(&subsys->dev);
+	return ret;
 }
 
 /*
@@ -1754,9 +1877,13 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 		return -EIO;
 	}
 
-	nvme_init_subnqn(ctrl, id);
-
 	if (!ctrl->identified) {
+		int i;
+
+		ret = nvme_init_subsystem(ctrl, id);
+		if (ret)
+			goto out_free;
+
 		/*
 		 * Check for quirks.  Quirk can depend on firmware version,
 		 * so, in principle, the set of quirks present can change
@@ -1765,9 +1892,6 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 		 * the device, but we'd have to make sure that the driver
 		 * behaves intelligently if the quirks change.
 		 */
-
-		int i;
-
 		for (i = 0; i < ARRAY_SIZE(core_quirks); i++) {
 			if (quirk_matches(id, &core_quirks[i]))
 				ctrl->quirks |= core_quirks[i].quirks;
@@ -1780,14 +1904,10 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 	}
 
 	ctrl->oacs = le16_to_cpu(id->oacs);
-	ctrl->vid = le16_to_cpu(id->vid);
 	ctrl->oncs = le16_to_cpup(&id->oncs);
 	atomic_set(&ctrl->abort_limit, id->acl + 1);
 	ctrl->vwc = id->vwc;
 	ctrl->cntlid = le16_to_cpup(&id->cntlid);
-	memcpy(ctrl->serial, id->sn, sizeof(id->sn));
-	memcpy(ctrl->model, id->mn, sizeof(id->mn));
-	memcpy(ctrl->firmware_rev, id->fr, sizeof(id->fr));
 	if (id->mdts)
 		max_hw_sectors = 1 << (id->mdts + page_shift - 9);
 	else
@@ -1990,9 +2110,9 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ctrl *ctrl = ns->ctrl;
-	int serial_len = sizeof(ctrl->serial);
-	int model_len = sizeof(ctrl->model);
+	struct nvme_subsystem *subsys = ns->ctrl->subsys;
+	int serial_len = sizeof(subsys->serial);
+	int model_len = sizeof(subsys->model);
 
 	if (!uuid_is_null(&ns->uuid))
 		return sprintf(buf, "uuid.%pU\n", &ns->uuid);
@@ -2003,15 +2123,16 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 	if (memchr_inv(ns->eui, 0, sizeof(ns->eui)))
 		return sprintf(buf, "eui.%8phN\n", ns->eui);
 
-	while (serial_len > 0 && (ctrl->serial[serial_len - 1] == ' ' ||
-				  ctrl->serial[serial_len - 1] == '\0'))
+	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
+				  subsys->serial[serial_len - 1] == '\0'))
 		serial_len--;
-	while (model_len > 0 && (ctrl->model[model_len - 1] == ' ' ||
-				 ctrl->model[model_len - 1] == '\0'))
+	while (model_len > 0 && (subsys->model[model_len - 1] == ' ' ||
+				 subsys->model[model_len - 1] == '\0'))
 		model_len--;
 
-	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", ctrl->vid,
-		serial_len, ctrl->serial, model_len, ctrl->model, ns->ns_id);
+	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
+		serial_len, subsys->serial, model_len, subsys->model,
+		ns->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
@@ -2097,10 +2218,15 @@ static ssize_t  field##_show(struct device *dev,				\
 			    struct device_attribute *attr, char *buf)		\
 {										\
         struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
-        return sprintf(buf, "%.*s\n", (int)sizeof(ctrl->field), ctrl->field);	\
+        return sprintf(buf, "%.*s\n",						\
+		(int)sizeof(ctrl->subsys->field), ctrl->subsys->field);		\
 }										\
 static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
 
+nvme_show_str_function(model);
+nvme_show_str_function(serial);
+nvme_show_str_function(firmware_rev);
+
 #define nvme_show_int_function(field)						\
 static ssize_t  field##_show(struct device *dev,				\
 			    struct device_attribute *attr, char *buf)		\
@@ -2110,9 +2236,6 @@ static ssize_t  field##_show(struct device *dev,				\
 }										\
 static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
 
-nvme_show_str_function(model);
-nvme_show_str_function(serial);
-nvme_show_str_function(firmware_rev);
 nvme_show_int_function(cntlid);
 
 static ssize_t nvme_sysfs_delete(struct device *dev,
@@ -2166,7 +2289,7 @@ static ssize_t nvme_sysfs_show_subsysnqn(struct device *dev,
 {
 	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
 
-	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subnqn);
+	return snprintf(buf, PAGE_SIZE, "%s\n", ctrl->subsys->subnqn);
 }
 static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
 
@@ -2675,11 +2798,21 @@ static void nvme_free_ctrl(struct device *dev)
 {
 	struct nvme_ctrl *ctrl =
 		container_of(dev, struct nvme_ctrl, ctrl_device);
+	struct nvme_subsystem *subsys = ctrl->subsys;
 
 	ida_simple_remove(&nvme_instance_ida, ctrl->instance);
 	ida_destroy(&ctrl->ns_ida);
 
+	if (subsys) {
+		mutex_lock(&subsys->lock);
+		list_del(&ctrl->subsys_entry);
+		mutex_unlock(&subsys->lock);
+	}
+
 	ctrl->ops->free_ctrl(ctrl);
+
+	if (subsys)
+		nvme_put_subsystem(subsys);
 }
 
 /*
@@ -2878,8 +3011,15 @@ int __init nvme_core_init(void)
 		goto unregister_chrdev;
 	}
 
+	nvme_subsys_class = class_create(THIS_MODULE, "nvme-subsystem");
+	if (IS_ERR(nvme_subsys_class)) {
+		result = PTR_ERR(nvme_subsys_class);
+		goto destroy_class;
+	}
 	return 0;
 
+destroy_class:
+	class_destroy(nvme_class);
 unregister_chrdev:
 	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 destroy_wq:
@@ -2889,6 +3029,8 @@ int __init nvme_core_init(void)
 
 void nvme_core_exit(void)
 {
+	ida_destroy(&nvme_subsystems_ida);
+	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
 	unregister_chrdev_region(nvme_chr_devt, NVME_MINORS);
 	destroy_workqueue(nvme_wq);
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 8bca36a46924..dcad66e37a8f 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -877,10 +877,10 @@ nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
 		goto out_unlock;
 	}
 
-	if (strcmp(ctrl->subnqn, opts->subsysnqn)) {
+	if (strcmp(ctrl->subsys->subnqn, opts->subsysnqn)) {
 		dev_warn(ctrl->device,
 			"controller returned incorrect NQN: \"%s\".\n",
-			ctrl->subnqn);
+			ctrl->subsys->subnqn);
 		up_read(&nvmf_transports_rwsem);
 		ctrl->ops->delete_ctrl(ctrl);
 		return ERR_PTR(-EINVAL);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1bb2bc165e54..6a5702072048 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -139,13 +139,12 @@ struct nvme_ctrl {
 	struct ida ns_ida;
 	struct work_struct reset_work;
 
+	struct nvme_subsystem *subsys;
+	struct list_head subsys_entry;
+
 	struct opal_dev *opal_dev;
 
 	char name[12];
-	char serial[20];
-	char model[40];
-	char firmware_rev[8];
-	char subnqn[NVMF_NQN_SIZE];
 	u16 cntlid;
 
 	u32 ctrl_config;
@@ -156,7 +155,6 @@ struct nvme_ctrl {
 	u32 page_size;
 	u32 max_hw_sectors;
 	u16 oncs;
-	u16 vid;
 	u16 oacs;
 	u16 nssa;
 	u16 nr_streams;
@@ -198,6 +196,24 @@ struct nvme_ctrl {
 	struct nvmf_ctrl_options *opts;
 };
 
+struct nvme_subsystem {
+	int			instance;
+	struct device		dev;
+	/*
+	 * Because we unregister the device on the last put we need
+	 * a separate refcount.
+	 */
+	struct kref		ref;
+	struct list_head	entry;
+	struct mutex		lock;
+	struct list_head	ctrls;
+	char			subnqn[NVMF_NQN_SIZE];
+	char			serial[20];
+	char			model[40];
+	char			firmware_rev[8];
+	u16			vendor_id;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:23 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:23 +0200
Subject: [PATCH 14/17] nvme: introduce a nvme_ns_ids structure
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-15-hch@lst.de>

This allows us to manage the various uniqueue namespace identifiers
together instead needing various variables and arguments.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 69 +++++++++++++++++++++++++++---------------------
 drivers/nvme/host/nvme.h | 14 +++++++---
 2 files changed, 49 insertions(+), 34 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index b3d468c77684..ab1a8022ead3 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -750,7 +750,7 @@ static int nvme_identify_ctrl(struct nvme_ctrl *dev, struct nvme_id_ctrl **id)
 }
 
 static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
-		u8 *eui64, u8 *nguid, uuid_t *uuid)
+		struct nvme_ns_ids *ids)
 {
 	struct nvme_command c = { };
 	int status;
@@ -786,7 +786,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_EUI64_LEN;
-			memcpy(eui64, data + pos + sizeof(*cur), len);
+			memcpy(ids->eui64, data + pos + sizeof(*cur), len);
 			break;
 		case NVME_NIDT_NGUID:
 			if (cur->nidl != NVME_NIDT_NGUID_LEN) {
@@ -796,7 +796,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_NGUID_LEN;
-			memcpy(nguid, data + pos + sizeof(*cur), len);
+			memcpy(ids->nguid, data + pos + sizeof(*cur), len);
 			break;
 		case NVME_NIDT_UUID:
 			if (cur->nidl != NVME_NIDT_UUID_LEN) {
@@ -806,7 +806,7 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, unsigned nsid,
 				goto free_data;
 			}
 			len = NVME_NIDT_UUID_LEN;
-			uuid_copy(uuid, data + pos + sizeof(*cur));
+			uuid_copy(&ids->uuid, data + pos + sizeof(*cur));
 			break;
 		default:
 			/* Skip unnkown types */
@@ -1138,22 +1138,31 @@ static void nvme_config_discard(struct nvme_ns *ns)
 }
 
 static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
-		struct nvme_id_ns *id, u8 *eui64, u8 *nguid, uuid_t *uuid)
+		struct nvme_id_ns *id, struct nvme_ns_ids *ids)
 {
+	memset(ids, 0, sizeof(*ids));
+
 	if (ctrl->vs >= NVME_VS(1, 1, 0))
-		memcpy(eui64, id->eui64, sizeof(id->eui64));
+		memcpy(ids->eui64, id->eui64, sizeof(id->eui64));
 	if (ctrl->vs >= NVME_VS(1, 2, 0))
-		memcpy(nguid, id->nguid, sizeof(id->nguid));
+		memcpy(ids->nguid, id->nguid, sizeof(id->nguid));
 	if (ctrl->vs >= NVME_VS(1, 3, 0)) {
 		 /* Don't treat error as fatal we potentially
 		  * already have a NGUID or EUI-64
 		  */
-		if (nvme_identify_ns_descs(ctrl, nsid, eui64, nguid, uuid))
+		if (nvme_identify_ns_descs(ctrl, nsid, ids))
 			dev_warn(ctrl->device,
 				 "%s: Identify Descriptors failed\n", __func__);
 	}
 }
 
+static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b)
+{
+	return uuid_equal(&a->uuid, &b->uuid) &&
+		memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) == 0 &&
+		memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) == 0;
+}
+
 static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
 {
 	struct nvme_ns *ns = disk->private_data;
@@ -1194,8 +1203,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 	struct nvme_ns *ns = disk->private_data;
 	struct nvme_ctrl *ctrl = ns->ctrl;
 	struct nvme_id_ns *id;
-	u8 eui64[8] = { 0 }, nguid[16] = { 0 };
-	uuid_t uuid = uuid_null;
+	struct nvme_ns_ids ids;
 	int ret = 0;
 
 	if (test_bit(NVME_NS_DEAD, &ns->flags)) {
@@ -1212,10 +1220,8 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		goto out;
 	}
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, eui64, nguid, &uuid);
-	if (!uuid_equal(&ns->uuid, &uuid) ||
-	    memcmp(&ns->nguid, &nguid, sizeof(ns->nguid)) ||
-	    memcmp(&ns->eui, &eui64, sizeof(ns->eui))) {
+	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ids);
+	if (!nvme_ns_ids_equal(&ns->ids, &ids)) {
 		dev_err(ctrl->device,
 			"identifiers changed for nsid %d\n", ns->ns_id);
 		ret = -ENODEV;
@@ -2110,18 +2116,19 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 	struct nvme_subsystem *subsys = ns->ctrl->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
 
-	if (!uuid_is_null(&ns->uuid))
-		return sprintf(buf, "uuid.%pU\n", &ns->uuid);
+	if (!uuid_is_null(&ids->uuid))
+		return sprintf(buf, "uuid.%pU\n", &ids->uuid);
 
-	if (memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
-		return sprintf(buf, "eui.%16phN\n", ns->nguid);
+	if (memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
+		return sprintf(buf, "eui.%16phN\n", ids->nguid);
 
-	if (memchr_inv(ns->eui, 0, sizeof(ns->eui)))
-		return sprintf(buf, "eui.%8phN\n", ns->eui);
+	if (memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
+		return sprintf(buf, "eui.%8phN\n", ids->eui64);
 
 	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
 				  subsys->serial[serial_len - 1] == '\0'))
@@ -2140,7 +2147,7 @@ static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
 			  char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->nguid);
+	return sprintf(buf, "%pU\n", ns->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
@@ -2148,16 +2155,17 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
 	 */
-	if (uuid_is_null(&ns->uuid)) {
+	if (uuid_is_null(&ids->uuid)) {
 		printk_ratelimited(KERN_WARNING
 				   "No UUID available providing old NGUID\n");
-		return sprintf(buf, "%pU\n", ns->nguid);
+		return sprintf(buf, "%pU\n", ids->nguid);
 	}
-	return sprintf(buf, "%pU\n", &ns->uuid);
+	return sprintf(buf, "%pU\n", &ids->uuid);
 }
 static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
 
@@ -2165,7 +2173,7 @@ static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->eui);
+	return sprintf(buf, "%8phd\n", ns->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
@@ -2191,18 +2199,19 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_ids *ids = &ns->ids;
 
 	if (a == &dev_attr_uuid.attr) {
-		if (uuid_is_null(&ns->uuid) ||
-		    !memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+		if (uuid_is_null(&ids->uuid) ||
+		    !memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
 			return 0;
 	}
 	if (a == &dev_attr_nguid.attr) {
-		if (!memchr_inv(ns->nguid, 0, sizeof(ns->nguid)))
+		if (!memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
 			return 0;
 	}
 	if (a == &dev_attr_eui.attr) {
-		if (!memchr_inv(ns->eui, 0, sizeof(ns->eui)))
+		if (!memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
 			return 0;
 	}
 	return a->mode;
@@ -2435,7 +2444,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, ns->eui, ns->nguid, &ns->uuid);
+	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ns->ids);
 
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 6a5702072048..efbf4dde6c87 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -214,6 +214,15 @@ struct nvme_subsystem {
 	u16			vendor_id;
 };
 
+/*
+ * Container structure for uniqueue namespace identifiers.
+ */
+struct nvme_ns_ids {
+	u8	eui64[8];
+	u8	nguid[16];
+	uuid_t	uuid;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
@@ -224,11 +233,8 @@ struct nvme_ns {
 	struct kref kref;
 	int instance;
 
-	u8 eui[8];
-	u8 nguid[16];
-	uuid_t uuid;
-
 	unsigned ns_id;
+	struct nvme_ns_ids ids;
 	int lba_shift;
 	u16 ms;
 	u16 sgs;
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:24 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:24 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-16-hch@lst.de>

Introduce a new struct nvme_ns_head that holds information about an actual
namespace, unlike struct nvme_ns, which only holds the per-controller
namespace information.  For private namespaces there is a 1:1 relation of
the two, but for shared namespaces this lets us discover all the paths to
it.  For now only the identifiers are moved to the new structure, but most
of the information in struct nvme_ns should eventually move over.

To allow lockless path lookup the list of nvme_ns structures per
nvme_ns_head is protected by SRCU, which requires freeing the nvme_ns
structure through call_srcu.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
Reviewed-by: Javier Gonz?lez <javier at cnexlabs.com>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c     | 190 +++++++++++++++++++++++++++++++++++++------
 drivers/nvme/host/lightnvm.c |  14 ++--
 drivers/nvme/host/nvme.h     |  21 ++++-
 3 files changed, 190 insertions(+), 35 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index ab1a8022ead3..1db26729bd89 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -245,6 +245,21 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 }
 EXPORT_SYMBOL_GPL(nvme_change_ctrl_state);
 
+static void nvme_free_ns_head(struct kref *ref)
+{
+	struct nvme_ns_head *head =
+		container_of(ref, struct nvme_ns_head, ref);
+
+	list_del_init(&head->entry);
+	cleanup_srcu_struct(&head->srcu);
+	kfree(head);
+}
+
+static void nvme_put_ns_head(struct nvme_ns_head *head)
+{
+	kref_put(&head->ref, nvme_free_ns_head);
+}
+
 static void nvme_free_ns(struct kref *kref)
 {
 	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
@@ -254,6 +269,7 @@ static void nvme_free_ns(struct kref *kref)
 
 	put_disk(ns->disk);
 	ida_simple_remove(&ns->ctrl->ns_ida, ns->instance);
+	nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
 	kfree(ns);
 }
@@ -389,7 +405,7 @@ static inline void nvme_setup_flush(struct nvme_ns *ns,
 {
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->common.opcode = nvme_cmd_flush;
-	cmnd->common.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->common.nsid = cpu_to_le32(ns->head->ns_id);
 }
 
 static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req,
@@ -420,7 +436,7 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req,
 
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->dsm.opcode = nvme_cmd_dsm;
-	cmnd->dsm.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->dsm.nsid = cpu_to_le32(ns->head->ns_id);
 	cmnd->dsm.nr = cpu_to_le32(segments - 1);
 	cmnd->dsm.attributes = cpu_to_le32(NVME_DSMGMT_AD);
 
@@ -459,7 +475,7 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns,
 
 	memset(cmnd, 0, sizeof(*cmnd));
 	cmnd->rw.opcode = (rq_data_dir(req) ? nvme_cmd_write : nvme_cmd_read);
-	cmnd->rw.nsid = cpu_to_le32(ns->ns_id);
+	cmnd->rw.nsid = cpu_to_le32(ns->head->ns_id);
 	cmnd->rw.slba = cpu_to_le64(nvme_block_nr(ns, blk_rq_pos(req)));
 	cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1);
 
@@ -940,7 +956,7 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 	memset(&c, 0, sizeof(c));
 	c.rw.opcode = io.opcode;
 	c.rw.flags = io.flags;
-	c.rw.nsid = cpu_to_le32(ns->ns_id);
+	c.rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c.rw.slba = cpu_to_le64(io.slba);
 	c.rw.length = cpu_to_le16(io.nblocks);
 	c.rw.control = cpu_to_le16(io.control);
@@ -1005,7 +1021,7 @@ static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
 	switch (cmd) {
 	case NVME_IOCTL_ID:
 		force_successful_syscall_return();
-		return ns->ns_id;
+		return ns->head->ns_id;
 	case NVME_IOCTL_ADMIN_CMD:
 		return nvme_user_cmd(ns->ctrl, NULL, (void __user *)arg);
 	case NVME_IOCTL_IO_CMD:
@@ -1156,6 +1172,13 @@ static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,
 	}
 }
 
+static bool nvme_ns_ids_valid(struct nvme_ns_ids *ids)
+{
+	return !uuid_is_null(&ids->uuid) ||
+		memchr_inv(ids->nguid, 0, sizeof(ids->nguid)) ||
+		memchr_inv(ids->eui64, 0, sizeof(ids->eui64));
+}
+
 static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b)
 {
 	return uuid_equal(&a->uuid, &b->uuid) &&
@@ -1211,7 +1234,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		return -ENODEV;
 	}
 
-	id = nvme_identify_ns(ctrl, ns->ns_id);
+	id = nvme_identify_ns(ctrl, ns->head->ns_id);
 	if (!id)
 		return -ENODEV;
 
@@ -1220,10 +1243,10 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		goto out;
 	}
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ids);
-	if (!nvme_ns_ids_equal(&ns->ids, &ids)) {
+	nvme_report_ns_ids(ctrl, ns->head->ns_id, id, &ids);
+	if (!nvme_ns_ids_equal(&ns->head->ids, &ids)) {
 		dev_err(ctrl->device,
-			"identifiers changed for nsid %d\n", ns->ns_id);
+			"identifiers changed for nsid %d\n", ns->head->ns_id);
 		ret = -ENODEV;
 	}
 
@@ -1264,7 +1287,7 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = op;
-	c.common.nsid = cpu_to_le32(ns->ns_id);
+	c.common.nsid = cpu_to_le32(ns->head->ns_id);
 	c.common.cdw10[0] = cpu_to_le32(cdw10);
 
 	return nvme_submit_sync_cmd(ns->queue, &c, data, 16);
@@ -1793,6 +1816,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	subsys->instance = ret;
 	kref_init(&subsys->ref);
 	INIT_LIST_HEAD(&subsys->ctrls);
+	INIT_LIST_HEAD(&subsys->nsheads);
 	nvme_init_subnqn(subsys, ctrl, id);
 	memcpy(subsys->serial, id->sn, sizeof(subsys->serial));
 	memcpy(subsys->model, id->mn, sizeof(subsys->model));
@@ -2116,7 +2140,7 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 	struct nvme_subsystem *subsys = ns->ctrl->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
@@ -2139,7 +2163,7 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 
 	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
 		serial_len, subsys->serial, model_len, subsys->model,
-		ns->ns_id);
+		ns->head->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
@@ -2147,7 +2171,7 @@ static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
 			  char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->ids.nguid);
+	return sprintf(buf, "%pU\n", ns->head->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
@@ -2155,7 +2179,7 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
@@ -2173,7 +2197,7 @@ static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->ids.eui64);
+	return sprintf(buf, "%8phd\n", ns->head->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
@@ -2181,7 +2205,7 @@ static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
 								char *buf)
 {
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%d\n", ns->ns_id);
+	return sprintf(buf, "%d\n", ns->head->ns_id);
 }
 static DEVICE_ATTR(nsid, S_IRUGO, nsid_show, NULL);
 
@@ -2199,7 +2223,7 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->ids;
+	struct nvme_ns_ids *ids = &ns->head->ids;
 
 	if (a == &dev_attr_uuid.attr) {
 		if (uuid_is_null(&ids->uuid) ||
@@ -2351,12 +2375,114 @@ static const struct attribute_group *nvme_dev_attr_groups[] = {
 	NULL,
 };
 
+static struct nvme_ns_head *__nvme_find_ns_head(struct nvme_subsystem *subsys,
+		unsigned nsid)
+{
+	struct nvme_ns_head *h;
+
+	lockdep_assert_held(&subsys->lock);
+
+	list_for_each_entry(h, &subsys->nsheads, entry) {
+		if (h->ns_id == nsid && kref_get_unless_zero(&h->ref))
+			return h;
+	}
+
+	return NULL;
+}
+
+static int __nvme_check_ids(struct nvme_subsystem *subsys,
+		struct nvme_ns_head *new)
+{
+	struct nvme_ns_head *h;
+
+	lockdep_assert_held(&subsys->lock);
+
+	list_for_each_entry(h, &subsys->nsheads, entry) {
+		if (nvme_ns_ids_valid(&new->ids) &&
+		    nvme_ns_ids_equal(&new->ids, &h->ids))
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
+		unsigned nsid, struct nvme_id_ns *id)
+{
+	struct nvme_ns_head *head;
+	int ret = -ENOMEM;
+
+	head = kzalloc(sizeof(*head), GFP_KERNEL);
+	if (!head)
+		goto out;
+
+	INIT_LIST_HEAD(&head->list);
+	head->ns_id = nsid;
+	init_srcu_struct(&head->srcu);
+	kref_init(&head->ref);
+
+	nvme_report_ns_ids(ctrl, nsid, id, &head->ids);
+
+	ret = __nvme_check_ids(ctrl->subsys, head);
+	if (ret) {
+		dev_err(ctrl->device,
+			"duplicate IDs for nsid %d\n", nsid);
+		goto out_free_head;
+	}
+
+	list_add_tail(&head->entry, &ctrl->subsys->nsheads);
+	return head;
+out_free_head:
+	cleanup_srcu_struct(&head->srcu);
+	kfree(head);
+out:
+	return ERR_PTR(ret);
+}
+
+static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
+		struct nvme_id_ns *id)
+{
+	struct nvme_ctrl *ctrl = ns->ctrl;
+	bool is_shared = id->nmic & (1 << 0);
+	struct nvme_ns_head *head = NULL;
+	int ret = 0;
+
+	mutex_lock(&ctrl->subsys->lock);
+	if (is_shared)
+		head = __nvme_find_ns_head(ctrl->subsys, nsid);
+	if (!head) {
+		head = nvme_alloc_ns_head(ctrl, nsid, id);
+		if (IS_ERR(head)) {
+			ret = PTR_ERR(head);
+			goto out_unlock;
+		}
+	} else {
+		struct nvme_ns_ids ids;
+
+		nvme_report_ns_ids(ctrl, nsid, id, &ids);
+		if (!nvme_ns_ids_equal(&head->ids, &ids)) {
+			dev_err(ctrl->device,
+				"IDs don't match for shared namespace %d\n",
+					nsid);
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+	}
+
+	list_add_tail(&ns->siblings, &head->list);
+	ns->head = head;
+
+out_unlock:
+	mutex_unlock(&ctrl->subsys->lock);
+	return ret;
+}
+
 static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 {
 	struct nvme_ns *nsa = container_of(a, struct nvme_ns, list);
 	struct nvme_ns *nsb = container_of(b, struct nvme_ns, list);
 
-	return nsa->ns_id - nsb->ns_id;
+	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
 static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
@@ -2365,13 +2491,13 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	mutex_lock(&ctrl->namespaces_mutex);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
-		if (ns->ns_id == nsid) {
+		if (ns->head->ns_id == nsid) {
 			if (!kref_get_unless_zero(&ns->kref))
 				continue;
 			ret = ns;
 			break;
 		}
-		if (ns->ns_id > nsid)
+		if (ns->head->ns_id > nsid)
 			break;
 	}
 	mutex_unlock(&ctrl->namespaces_mutex);
@@ -2386,7 +2512,7 @@ static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns)
 	if (!ctrl->nr_streams)
 		return 0;
 
-	ret = nvme_get_stream_params(ctrl, &s, ns->ns_id);
+	ret = nvme_get_stream_params(ctrl, &s, ns->head->ns_id);
 	if (ret)
 		return ret;
 
@@ -2428,7 +2554,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	ns->ctrl = ctrl;
 
 	kref_init(&ns->kref);
-	ns->ns_id = nsid;
 	ns->lba_shift = 9; /* set to a default value for 512 until disk is validated */
 
 	blk_queue_logical_block_size(ns->queue, 1 << ns->lba_shift);
@@ -2444,18 +2569,19 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	if (id->ncap == 0)
 		goto out_free_id;
 
-	nvme_report_ns_ids(ctrl, ns->ns_id, id, &ns->ids);
+	if (nvme_init_ns_head(ns, nsid, id))
+		goto out_free_id;
 
 	if ((ctrl->quirks & NVME_QUIRK_LIGHTNVM) && id->vs[0] == 0x1) {
 		if (nvme_nvm_register(ns, disk_name, node)) {
 			dev_warn(ctrl->device, "LightNVM init failure\n");
-			goto out_free_id;
+			goto out_unlink_ns;
 		}
 	}
 
 	disk = alloc_disk_node(0, node);
 	if (!disk)
-		goto out_free_id;
+		goto out_unlink_ns;
 
 	disk->fops = &nvme_fops;
 	disk->private_data = ns;
@@ -2483,6 +2609,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
 	return;
+ out_unlink_ns:
+	mutex_lock(&ctrl->subsys->lock);
+	list_del_rcu(&ns->siblings);
+	mutex_unlock(&ctrl->subsys->lock);
  out_free_id:
 	kfree(id);
  out_free_queue:
@@ -2495,6 +2625,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 static void nvme_ns_remove(struct nvme_ns *ns)
 {
+	struct nvme_ns_head *head = ns->head;
+
 	if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags))
 		return;
 
@@ -2509,10 +2641,16 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 		blk_cleanup_queue(ns->queue);
 	}
 
+	mutex_lock(&ns->ctrl->subsys->lock);
+	if (head)
+		list_del_rcu(&ns->siblings);
+	mutex_unlock(&ns->ctrl->subsys->lock);
+
 	mutex_lock(&ns->ctrl->namespaces_mutex);
 	list_del_init(&ns->list);
 	mutex_unlock(&ns->ctrl->namespaces_mutex);
 
+	synchronize_srcu(&head->srcu);
 	nvme_put_ns(ns);
 }
 
@@ -2535,7 +2673,7 @@ static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
 	struct nvme_ns *ns, *next;
 
 	list_for_each_entry_safe(ns, next, &ctrl->namespaces, list) {
-		if (ns->ns_id > nsid)
+		if (ns->head->ns_id > nsid)
 			nvme_ns_remove(ns);
 	}
 }
diff --git a/drivers/nvme/host/lightnvm.c b/drivers/nvme/host/lightnvm.c
index 1f79e3f141e6..44e46276319c 100644
--- a/drivers/nvme/host/lightnvm.c
+++ b/drivers/nvme/host/lightnvm.c
@@ -305,7 +305,7 @@ static int nvme_nvm_identity(struct nvm_dev *nvmdev, struct nvm_id *nvm_id)
 	int ret;
 
 	c.identity.opcode = nvme_nvm_admin_identity;
-	c.identity.nsid = cpu_to_le32(ns->ns_id);
+	c.identity.nsid = cpu_to_le32(ns->head->ns_id);
 	c.identity.chnl_off = 0;
 
 	nvme_nvm_id = kmalloc(sizeof(struct nvme_nvm_id), GFP_KERNEL);
@@ -344,7 +344,7 @@ static int nvme_nvm_get_l2p_tbl(struct nvm_dev *nvmdev, u64 slba, u32 nlb,
 	int ret = 0;
 
 	c.l2p.opcode = nvme_nvm_admin_get_l2p_tbl;
-	c.l2p.nsid = cpu_to_le32(ns->ns_id);
+	c.l2p.nsid = cpu_to_le32(ns->head->ns_id);
 	entries = kmalloc(len, GFP_KERNEL);
 	if (!entries)
 		return -ENOMEM;
@@ -402,7 +402,7 @@ static int nvme_nvm_get_bb_tbl(struct nvm_dev *nvmdev, struct ppa_addr ppa,
 	int ret = 0;
 
 	c.get_bb.opcode = nvme_nvm_admin_get_bb_tbl;
-	c.get_bb.nsid = cpu_to_le32(ns->ns_id);
+	c.get_bb.nsid = cpu_to_le32(ns->head->ns_id);
 	c.get_bb.spba = cpu_to_le64(ppa.ppa);
 
 	bb_tbl = kzalloc(tblsz, GFP_KERNEL);
@@ -452,7 +452,7 @@ static int nvme_nvm_set_bb_tbl(struct nvm_dev *nvmdev, struct ppa_addr *ppas,
 	int ret = 0;
 
 	c.set_bb.opcode = nvme_nvm_admin_set_bb_tbl;
-	c.set_bb.nsid = cpu_to_le32(ns->ns_id);
+	c.set_bb.nsid = cpu_to_le32(ns->head->ns_id);
 	c.set_bb.spba = cpu_to_le64(ppas->ppa);
 	c.set_bb.nlb = cpu_to_le16(nr_ppas - 1);
 	c.set_bb.value = type;
@@ -469,7 +469,7 @@ static inline void nvme_nvm_rqtocmd(struct nvm_rq *rqd, struct nvme_ns *ns,
 				    struct nvme_nvm_command *c)
 {
 	c->ph_rw.opcode = rqd->opcode;
-	c->ph_rw.nsid = cpu_to_le32(ns->ns_id);
+	c->ph_rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c->ph_rw.spba = cpu_to_le64(rqd->ppa_addr.ppa);
 	c->ph_rw.metadata = cpu_to_le64(rqd->dma_meta_list);
 	c->ph_rw.control = cpu_to_le16(rqd->flags);
@@ -691,7 +691,7 @@ static int nvme_nvm_submit_vio(struct nvme_ns *ns,
 
 	memset(&c, 0, sizeof(c));
 	c.ph_rw.opcode = vio.opcode;
-	c.ph_rw.nsid = cpu_to_le32(ns->ns_id);
+	c.ph_rw.nsid = cpu_to_le32(ns->head->ns_id);
 	c.ph_rw.control = cpu_to_le16(vio.control);
 	c.ph_rw.length = cpu_to_le16(vio.nppas);
 
@@ -728,7 +728,7 @@ static int nvme_nvm_user_vcmd(struct nvme_ns *ns, int admin,
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = vcmd.opcode;
-	c.common.nsid = cpu_to_le32(ns->ns_id);
+	c.common.nsid = cpu_to_le32(ns->head->ns_id);
 	c.common.cdw2[0] = cpu_to_le32(vcmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(vcmd.cdw3);
 	/* cdw11-12 */
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index efbf4dde6c87..849413def126 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -207,6 +207,7 @@ struct nvme_subsystem {
 	struct list_head	entry;
 	struct mutex		lock;
 	struct list_head	ctrls;
+	struct list_head	nsheads;
 	char			subnqn[NVMF_NQN_SIZE];
 	char			serial[20];
 	char			model[40];
@@ -223,18 +224,34 @@ struct nvme_ns_ids {
 	uuid_t	uuid;
 };
 
+/*
+ * Anchor structure for namespaces.  There is one for each namespace in a
+ * NVMe subsystem that any of our controllers can see, and the namespace
+ * structure for each controller is chained of it.  For private namespaces
+ * there is a 1:1 relation to our namespace structures, that is ->list
+ * only ever has a single entry for private namespaces.
+ */
+struct nvme_ns_head {
+	struct list_head	list;
+	struct srcu_struct      srcu;
+	unsigned		ns_id;
+	struct nvme_ns_ids	ids;
+	struct list_head	entry;
+	struct kref		ref;
+};
+
 struct nvme_ns {
 	struct list_head list;
 
 	struct nvme_ctrl *ctrl;
 	struct request_queue *queue;
 	struct gendisk *disk;
+	struct list_head siblings;
 	struct nvm_dev *ndev;
 	struct kref kref;
+	struct nvme_ns_head *head;
 	int instance;
 
-	unsigned ns_id;
-	struct nvme_ns_ids ids;
 	int lba_shift;
 	u16 ms;
 	u16 sgs;
-- 
2.14.2



From hch at lst.de  Mon Oct 23 07:51:26 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 16:51:26 +0200
Subject: [PATCH 17/17] nvme: also expose the namespace identification sysfs
 files for mpath nodes
In-Reply-To: <20171023145126.2471-1-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
Message-ID: <20171023145126.2471-18-hch@lst.de>

We do this by adding a helper that returns the ns_head for a device that
can belong to either the per-controller or per-subsystem block device
nodes, and otherwise reuse all the existing code.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Keith Busch <keith.busch at intel.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
---
 drivers/nvme/host/core.c | 69 +++++++++++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 27 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 22c06cd3bef0..334735db90c8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -76,6 +76,7 @@ static DEFINE_IDA(nvme_instance_ida);
 static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 static struct class *nvme_subsys_class;
+static const struct attribute_group nvme_ns_id_attr_group;
 
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
 {
@@ -329,6 +330,8 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	sysfs_remove_group(&disk_to_dev(head->disk)->kobj,
+			   &nvme_ns_id_attr_group);
 	del_gendisk(head->disk);
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
@@ -1925,7 +1928,7 @@ static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
 static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 {
 	struct nvme_subsystem *subsys, *found;
-	int ret = -ENOMEM;
+	int ret;
 
 	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
 	if (!subsys)
@@ -2267,12 +2270,22 @@ static ssize_t nvme_sysfs_rescan(struct device *dev,
 }
 static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan);
 
+static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
+{
+	struct gendisk *disk = dev_to_disk(dev);
+
+	if (disk->fops == &nvme_fops)
+		return nvme_get_ns_from_dev(dev)->head;
+	else
+		return disk->private_data;
+}
+
 static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
-	struct nvme_subsystem *subsys = ns->ctrl->subsys;
+	struct nvme_ns_head *head = dev_to_ns_head(dev);
+	struct nvme_ns_ids *ids = &head->ids;
+	struct nvme_subsystem *subsys = head->subsys;
 	int serial_len = sizeof(subsys->serial);
 	int model_len = sizeof(subsys->model);
 
@@ -2294,23 +2307,21 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
 
 	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
 		serial_len, subsys->serial, model_len, subsys->model,
-		ns->head->ns_id);
+		head->ns_id);
 }
 static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
 
 static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
-			  char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%pU\n", ns->head->ids.nguid);
+	return sprintf(buf, "%pU\n", dev_to_ns_head(dev)->ids.nguid);
 }
 static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
 
 static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
 	/* For backward compatibility expose the NGUID to userspace if
 	 * we have no UUID set
@@ -2325,22 +2336,20 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
 
 static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%8phd\n", ns->head->ids.eui64);
+	return sprintf(buf, "%8phd\n", dev_to_ns_head(dev)->ids.eui64);
 }
 static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
 
 static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
-								char *buf)
+		char *buf)
 {
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	return sprintf(buf, "%d\n", ns->head->ns_id);
+	return sprintf(buf, "%d\n", dev_to_ns_head(dev)->ns_id);
 }
 static DEVICE_ATTR(nsid, S_IRUGO, nsid_show, NULL);
 
-static struct attribute *nvme_ns_attrs[] = {
+static struct attribute *nvme_ns_id_attrs[] = {
 	&dev_attr_wwid.attr,
 	&dev_attr_uuid.attr,
 	&dev_attr_nguid.attr,
@@ -2349,12 +2358,11 @@ static struct attribute *nvme_ns_attrs[] = {
 	NULL,
 };
 
-static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
+static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
 	struct device *dev = container_of(kobj, struct device, kobj);
-	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
-	struct nvme_ns_ids *ids = &ns->head->ids;
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
 	if (a == &dev_attr_uuid.attr) {
 		if (uuid_is_null(&ids->uuid) ||
@@ -2372,9 +2380,9 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 	return a->mode;
 }
 
-static const struct attribute_group nvme_ns_attr_group = {
-	.attrs		= nvme_ns_attrs,
-	.is_visible	= nvme_ns_attrs_are_visible,
+static const struct attribute_group nvme_ns_id_attr_group = {
+	.attrs		= nvme_ns_id_attrs,
+	.is_visible	= nvme_ns_id_attrs_are_visible,
 };
 
 #define nvme_show_str_function(field)						\
@@ -2883,15 +2891,20 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	device_add_disk(ctrl->device, ns->disk);
 	if (sysfs_create_group(&disk_to_dev(ns->disk)->kobj,
-					&nvme_ns_attr_group))
+					&nvme_ns_id_attr_group))
 		pr_warn("%s: failed to create sysfs group for identification\n",
 			ns->disk->disk_name);
 	if (ns->ndev && nvme_nvm_register_sysfs(ns))
 		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
 			ns->disk->disk_name);
 
-	if (new)
+	if (new) {
 		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
+		if (sysfs_create_group(&disk_to_dev(ns->head->disk)->kobj,
+				&nvme_ns_id_attr_group))
+			pr_warn("%s: failed to create sysfs group for identification\n",
+				ns->head->disk->disk_name);
+	}
 
 	return;
  out_unlink_ns:
@@ -2916,6 +2929,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		if (blk_get_integrity(ns->disk))
 			blk_integrity_unregister(ns->disk);
+		sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
+				   &nvme_ns_id_attr_group);
 		if (ns->ndev)
 			nvme_nvm_unregister_sysfs(ns);
 		del_gendisk(ns->disk);
-- 
2.14.2



From keith.busch at intel.com  Mon Oct 23 08:01:33 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 23 Oct 2017 09:01:33 -0600
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <20171021080906.GE20906@lst.de>
References: <20171020223746.31287-1-keith.busch@intel.com>
 <20171021080906.GE20906@lst.de>
Message-ID: <20171023150133.GD30713@localhost.localdomain>

On Sat, Oct 21, 2017 at 10:09:06AM +0200, Christoph Hellwig wrote:
> On Fri, Oct 20, 2017 at 04:37:46PM -0600, Keith Busch wrote:
> > This will give udev a chance to handle asynchronous event notification
> > and clear the log to unmask future events of the same type. Since the
> > core driver allows only one AEN request at a time, we can only have one
> > possible oustanding uevent to send.
> 
> I don't want to encode that in the userspace ABI.  With all kinds of
> new AEN uses coming up we might as well want to go for multiple AERs
> in the future.

Okay, that's certainly possible to do. The motivation for just one active
AEN was just to avoid having a second method for tracking outstanding
command tags.

> > This implementation saves the last
> > AEN result from the IRQ handler, and sends the uevent change notification
> > when the AEN work is rescheduled.
> > 
> > AEN events that the kernel handles directly will not create a uevent. Such
> > events will stop being sent to the user as new handlers are added to
> > the kernel.
> 
> I think we need an explicit whitelist of the ones we want to expose
> to userspace.  Otherwise we'll get all kinds of complaints later
> that we'd take something away and break the ABI.

Oh, I was thinking of this differently. The ABI proposed here is saying
all unhandled events are sent to the user, and the kernel has the right
to handle these in the future as the need arises; user space can't
depend on seeing an event if the kernel should handle it, but it gets to
know about all events the kernel didn't handle.

While this changes user visible events when new driver handling is
added, this is forward compatible with new events that aren't defined
yet.

I'm not particularly concerned about it either way. I just thought it
was easier treating the kernel handling as a blacklist to suppress
notification rather than have a whitelist.


From sagi at grimberg.me  Mon Oct 23 08:25:42 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 18:25:42 +0300
Subject: [PATCH 2/5] nvme-rdma: rework create_ctrl error flow
In-Reply-To: <20171023143255.GB25448@lst.de>
References: <1508324864-23087-1-git-send-email-sagi@grimberg.me>
 <1508324864-23087-3-git-send-email-sagi@grimberg.me>
 <20171023143255.GB25448@lst.de>
Message-ID: <9023b292-f403-f9fd-40cf-daa82fa60942@grimberg.me>


>> Now that we have nvme_init/uninit_ctrl doing the right thing, we
>> can fix this by making the error sequence sane and simple (just
>> undo what we did until the error).
>>
>> Also, allocate ctrl->queues before calling nvme_init_ctrl, just
>> to separate the driver private resource allocation and the core
>> resource allocations.
> 
> Once we did the init we'll have to do a put and not just free it.
> That will become even more important when using a struct device for
> refcounting the controller.

:(

I want to get to a point where the transports allocate everything they
need in their private struct, and then do nvme common things. This
asymmetric behavior is going hard to keep correct once got right.


From sagi at grimberg.me  Mon Oct 23 08:32:47 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 18:32:47 +0300
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171023145126.2471-17-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
Message-ID: <68df1af0-7cc8-febc-e2f6-85251f120a76@grimberg.me>


> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 7735571ffc9a..bbece5edabff 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -1050,6 +1050,8 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
>   {
>   	struct nvme_queue *nvmeq = hctx->driver_data;
>   
> +	printk_ratelimited("%s: called\n", __func__);
> +

This must be a left-over...


From Jeff.Lien at wdc.com  Mon Oct 23 08:43:36 2017
From: Jeff.Lien at wdc.com (Jeffrey Lien)
Date: Mon, 23 Oct 2017 15:43:36 +0000
Subject: NVMe-CLI intel-nvme.c RHEL 6.9 Compile Errors
Message-ID: <MWHPR04MB102554E84EE6D1CC8ADBF47EEA460@MWHPR04MB1025.namprd04.prod.outlook.com>

Keith, 
We noticed these compile errors when trying to build the latest version of the NVME CLI on RHEL 6.9 and SLES 11.4:
[ddtest at gfox3-rh69 nvme-cli-kb]$ make
NVME_VERSION = 1.4
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c argconfig.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c suffix.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c parser.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-print.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-ioctl.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-lightnvm.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c fabrics.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c json.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c plugin.c
cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c intel-nvme.c
cc1: warnings being treated as errors
intel-nvme.c: In function \u2018get_internal_log\u2019:
intel-nvme.c:782: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
intel-nvme.c:785: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
intel-nvme.c:741: note: initialized from here
intel-nvme.c:643: error: \u2018ad\u2019 may be used uninitialized in this function
intel-nvme.c:645: error: \u2018ehdr\u2019 may be used uninitialized in this function
make: *** [intel-nvme.o] Error 1

If I checkout the v1.3 tag, it builds fine, but then our latest wdc plugin changes aren't included.  Could  you (or whoever the intel plugin maintainer is) look into fixing these errors?   Thanks


Jeff Lien
Linux Device Driver Development
Device Host Apps and Drivers
jeff.lien at wdc.com
o: 507-322-2416 (ext. 23-2416)
m: 507-273-9124








From scott.bauer at intel.com  Mon Oct 23 08:48:45 2017
From: scott.bauer at intel.com (Scott Bauer)
Date: Mon, 23 Oct 2017 09:48:45 -0600
Subject: NVMe-CLI intel-nvme.c RHEL 6.9 Compile Errors
In-Reply-To: <MWHPR04MB102554E84EE6D1CC8ADBF47EEA460@MWHPR04MB1025.namprd04.prod.outlook.com>
References: <MWHPR04MB102554E84EE6D1CC8ADBF47EEA460@MWHPR04MB1025.namprd04.prod.outlook.com>
Message-ID: <20171023154844.bzg5crhusqehxrho@sbauer-Z170X-UD5>

On Mon, Oct 23, 2017 at 03:43:36PM +0000, Jeffrey Lien wrote:
> Keith, 
> We noticed these compile errors when trying to build the latest version of the NVME CLI on RHEL 6.9 and SLES 11.4:
> [ddtest at gfox3-rh69 nvme-cli-kb]$ make
> NVME_VERSION = 1.4
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c argconfig.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c suffix.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c parser.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-print.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-ioctl.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-lightnvm.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c fabrics.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c json.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c plugin.c
> cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c intel-nvme.c
> cc1: warnings being treated as errors
> intel-nvme.c: In function \u2018get_internal_log\u2019:
> intel-nvme.c:782: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
> intel-nvme.c:785: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
> intel-nvme.c:741: note: initialized from here
> intel-nvme.c:643: error: \u2018ad\u2019 may be used uninitialized in this function
> intel-nvme.c:645: error: \u2018ehdr\u2019 may be used uninitialized in this function
> make: *** [intel-nvme.o] Error 1
> 

If you need to build it right now while I fix it you can add -fno-strict-aliasing.

As for the actual error, I think the c-spec allows for a char type to be aliased to
any other type without invoking undefiened behavior. I'll try and find that in the spec
to verify. What GCC version does RHEL 6.9 have?



From hch at lst.de  Mon Oct 23 09:57:20 2017
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 23 Oct 2017 18:57:20 +0200
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <68df1af0-7cc8-febc-e2f6-85251f120a76@grimberg.me>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
 <68df1af0-7cc8-febc-e2f6-85251f120a76@grimberg.me>
Message-ID: <20171023165720.GA28944@lst.de>

On Mon, Oct 23, 2017 at 06:32:47PM +0300, Sagi Grimberg wrote:
>>   	struct nvme_queue *nvmeq = hctx->driver_data;
>>   +	printk_ratelimited("%s: called\n", __func__);
>> +
>
> This must be a left-over...

Indeed, it is a left-over debug statement..


From sagi at grimberg.me  Mon Oct 23 10:43:13 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 23 Oct 2017 20:43:13 +0300
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171023145126.2471-17-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
Message-ID: <b975ad9e-a40a-0e89-1a8d-fc9e417ff681@grimberg.me>


>   static inline bool nvme_req_needs_retry(struct request *req)
>   {
>   	if (blk_noretry_request(req))
> @@ -143,6 +204,11 @@ static inline bool nvme_req_needs_retry(struct request *req)
>   void nvme_complete_rq(struct request *req)
>   {
>   	if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
> +		if (nvme_req_needs_failover(req)) {
> +			nvme_failover_req(req);
> +			return;
> +		}
> +
>   		nvme_req(req)->retries++;
>   		blk_mq_requeue_request(req, true);
>   		return;

Nit, consider having the !nvme_req_needs_failover() case in an else
clause, just a suggestion though.



> @@ -171,6 +237,18 @@ void nvme_cancel_request(struct request *req, void *data, bool reserved)
>   }
>   EXPORT_SYMBOL_GPL(nvme_cancel_request);

Question,

Does the statement in nvme_cancel_request:

         if (blk_queue_dying(req->q))
                 status |= NVME_SC_DNR;

still holds?


From keith.busch at intel.com  Mon Oct 23 12:27:58 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 23 Oct 2017 13:27:58 -0600
Subject: NVMe-CLI intel-nvme.c RHEL 6.9 Compile Errors
In-Reply-To: <20171023154844.bzg5crhusqehxrho@sbauer-Z170X-UD5>
References: <MWHPR04MB102554E84EE6D1CC8ADBF47EEA460@MWHPR04MB1025.namprd04.prod.outlook.com>
 <20171023154844.bzg5crhusqehxrho@sbauer-Z170X-UD5>
Message-ID: <20171023192758.GF30713@localhost.localdomain>

On Mon, Oct 23, 2017 at 09:48:45AM -0600, Scott Bauer wrote:
> On Mon, Oct 23, 2017 at 03:43:36PM +0000, Jeffrey Lien wrote:
> > Keith, 
> > We noticed these compile errors when trying to build the latest version of the NVME CLI on RHEL 6.9 and SLES 11.4:
> > [ddtest at gfox3-rh69 nvme-cli-kb]$ make
> > NVME_VERSION = 1.4
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c argconfig.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c suffix.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c parser.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-print.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-ioctl.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c nvme-lightnvm.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c fabrics.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c json.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c plugin.c
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c intel-nvme.c
> > cc1: warnings being treated as errors
> > intel-nvme.c: In function \u2018get_internal_log\u2019:
> > intel-nvme.c:782: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
> > intel-nvme.c:785: error: dereferencing pointer \u2018ehdr\u2019 does break strict-aliasing rules
> > intel-nvme.c:741: note: initialized from here
> > intel-nvme.c:643: error: \u2018ad\u2019 may be used uninitialized in this function
> > intel-nvme.c:645: error: \u2018ehdr\u2019 may be used uninitialized in this function
> > make: *** [intel-nvme.o] Error 1
> > 
> 
> If you need to build it right now while I fix it you can add -fno-strict-aliasing.
> 
> As for the actual error, I think the c-spec allows for a char type to be aliased to
> any other type without invoking undefiened behavior. I'll try and find that in the spec
> to verify. What GCC version does RHEL 6.9 have?

Some kind of weird casting rules. Should be fixed now.


From Jeff.Lien at wdc.com  Mon Oct 23 12:34:36 2017
From: Jeff.Lien at wdc.com (Jeffrey Lien)
Date: Mon, 23 Oct 2017 19:34:36 +0000
Subject: NVMe-CLI intel-nvme.c RHEL 6.9 Compile Errors
In-Reply-To: <20171023192758.GF30713@localhost.localdomain>
References: <MWHPR04MB102554E84EE6D1CC8ADBF47EEA460@MWHPR04MB1025.namprd04.prod.outlook.com>
 <20171023154844.bzg5crhusqehxrho@sbauer-Z170X-UD5>
 <20171023192758.GF30713@localhost.localdomain>
Message-ID: <MWHPR04MB10256C932AEB9B69D077F417EA460@MWHPR04MB1025.namprd04.prod.outlook.com>

Thanks Keith.  Just verified on my Rhel 6.9 machine - no more compile errors.  


Jeff Lien

-----Original Message-----
From: Keith Busch [mailto:keith.busch at intel.com] 
Sent: Monday, October 23, 2017 2:28 PM
To: Scott Bauer
Cc: Jeffrey Lien; linux-nvme at lists.infradead.org; David Darrington
Subject: Re: NVMe-CLI intel-nvme.c RHEL 6.9 Compile Errors

On Mon, Oct 23, 2017 at 09:48:45AM -0600, Scott Bauer wrote:
> On Mon, Oct 23, 2017 at 03:43:36PM +0000, Jeffrey Lien wrote:
> > Keith,
> > We noticed these compile errors when trying to build the latest version of the NVME CLI on RHEL 6.9 and SLES 11.4:
> > [ddtest at gfox3-rh69 nvme-cli-kb]$ make NVME_VERSION = 1.4 cc 
> > -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c argconfig.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c suffix.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c parser.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c nvme-print.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c nvme-ioctl.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c nvme-lightnvm.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c fabrics.c cc -D_GNU_SOURCE 
> > -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c json.c cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ 
> > -O2 -g -Wall -Werror -std=gnu99 -DNVME_VERSION='"1.4"' -c plugin.c 
> > cc -D_GNU_SOURCE -D__CHECK_ENDIAN__ -O2 -g -Wall -Werror -std=gnu99 
> > -DNVME_VERSION='"1.4"' -c intel-nvme.c
> > cc1: warnings being treated as errors
> > intel-nvme.c: In function \u2018get_internal_log\u2019:
> > intel-nvme.c:782: error: dereferencing pointer \u2018ehdr\u2019 does 
> > break strict-aliasing rules
> > intel-nvme.c:785: error: dereferencing pointer \u2018ehdr\u2019 does 
> > break strict-aliasing rules
> > intel-nvme.c:741: note: initialized from here
> > intel-nvme.c:643: error: \u2018ad\u2019 may be used uninitialized in 
> > this function
> > intel-nvme.c:645: error: \u2018ehdr\u2019 may be used uninitialized 
> > in this function
> > make: *** [intel-nvme.o] Error 1
> > 
> 
> If you need to build it right now while I fix it you can add -fno-strict-aliasing.
> 
> As for the actual error, I think the c-spec allows for a char type to 
> be aliased to any other type without invoking undefiened behavior. 
> I'll try and find that in the spec to verify. What GCC version does RHEL 6.9 have?

Some kind of weird casting rules. Should be fixed now.


From jsmart2021 at gmail.com  Mon Oct 23 15:03:57 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon, 23 Oct 2017 15:03:57 -0700
Subject: [PATCH v3] nvmet: Fix fatal_err_work deadlock
Message-ID: <20171023220357.11106-1-jsmart2021@gmail.com>

Below is  a stack trace for an issue that was reported.

What's happening is that the nvmet layer had it's controller kato
timeout fire, which causes it to schedule its fatal error handler
via the fatal_err_work element. The error handler is invoked, which
calls the transport delete_ctrl() entry point, and as the transport
tears down the controller, nvmet_sq_destroy ends up doing the final
put on the ctlr causing it to enter its free routine. The ctlr free
routine does a cancel_work_sync() on fatal_err_work element, which
then does a flush_work and wait_for_completion. But, as the wait is
in the context of the work element being flushed, its in a catch-22
and the thread hangs.

[  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  326.909832] nvmet: ctrl 1 fatal error occurred!
[  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
x114 flg x2
[  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
seconds.
[  494.589638]       Not tainted 4.14.0-rc1.James+ #1
[  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  494.603718] kworker/0:2     D    0   243      2 0x80000000
[  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
[  494.616447] Call Trace:
[  494.619177]  __schedule+0x28d/0x890
[  494.623070]  schedule+0x36/0x80
[  494.626571]  schedule_timeout+0x1dd/0x300
[  494.631044]  ? dequeue_task_fair+0x592/0x840
[  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
[  494.640756]  wait_for_completion+0x121/0x180
[  494.645521]  ? wake_up_q+0x80/0x80
[  494.649315]  flush_work+0x11d/0x1a0
[  494.653206]  ? wake_up_worker+0x30/0x30
[  494.657484]  __cancel_work_timer+0x10b/0x190
[  494.662249]  cancel_work_sync+0x10/0x20
[  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
[  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
[  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
[  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
[  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
[  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
[  494.701589]  process_one_work+0x149/0x360
[  494.706064]  worker_thread+0x4d/0x3c0
[  494.710148]  kthread+0x109/0x140
[  494.713751]  ? rescuer_thread+0x380/0x380
[  494.718214]  ? kthread_park+0x60/0x60

Correct by using flush_work() rather than cancel_work_sync().

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v2:
  convert cancel_work_sync() to flush_work() for fatal_err_work
v3:
  remove the addition of a new workq context. not needed as
  flush_work supports being called within the context

 drivers/nvme/target/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index da088293eafc..009eaca1c006 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -865,7 +865,7 @@ static void nvmet_ctrl_free(struct kref *ref)
 	mutex_unlock(&subsys->lock);
 
 	flush_work(&ctrl->async_event_work);
-	cancel_work_sync(&ctrl->fatal_err_work);
+	flush_work(&ctrl->fatal_err_work);
 
 	ida_simple_remove(&cntlid_ida, ctrl->cntlid);
 	nvmet_subsys_put(subsys);
-- 
2.13.1



From jsmart2021 at gmail.com  Mon Oct 23 15:11:36 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Mon, 23 Oct 2017 15:11:36 -0700
Subject: [PATCH] nvme_fc: remove NVME_FC_MAX_SEGMENTS
Message-ID: <20171023221136.23216-1-jsmart2021@gmail.com>

The define is an arbitrary limit to the io size on the initiator,
capping the io to 1MB-4KB.

Remove the define from the transport. I/O size will solely be limited
by the LLDD sg limits.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c         | 6 ++----
 include/linux/nvme-fc-driver.h | 2 --
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index c60129949377..bfb688ba28db 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2644,7 +2644,6 @@ static int
 nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 {
 	struct nvmf_ctrl_options *opts = ctrl->ctrl.opts;
-	u32 segs;
 	int ret;
 	bool changed;
 
@@ -2698,9 +2697,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	if (ret)
 		goto out_disconnect_admin_queue;
 
-	segs = min_t(u32, NVME_FC_MAX_SEGMENTS,
-			ctrl->lport->ops->max_sgl_segments);
-	ctrl->ctrl.max_hw_sectors = (segs - 1) << (PAGE_SHIFT - 9);
+	ctrl->ctrl.max_hw_sectors =
+		(ctrl->lport->ops->max_sgl_segments - 1) << (PAGE_SHIFT - 9);
 
 	ret = nvme_init_identify(&ctrl->ctrl);
 	if (ret)
diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
index 3de268bf86bf..496ff759f84c 100644
--- a/include/linux/nvme-fc-driver.h
+++ b/include/linux/nvme-fc-driver.h
@@ -105,8 +105,6 @@ enum nvmefc_fcp_datadir {
 };
 
 
-#define NVME_FC_MAX_SEGMENTS		256
-
 /**
  * struct nvmefc_fcp_req - Request structure passed from NVME-FC transport
  *                         to LLDD in order to perform a NVME FCP IO operation.
-- 
2.13.1



From hare at suse.de  Tue Oct 24 00:05:58 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:05:58 +0200
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171023145126.2471-4-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-4-hch@lst.de>
Message-ID: <330023d9-b2de-04da-01c5-4d71e910f332@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This helper allows reinserting a bio into a new queue without much
> overhead, but requires all queue limits to be the same for the upper
> and lower queues, and it does not provide any recursion preventions.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>  block/blk-core.c       | 34 ++++++++++++++++++++++++++++++++++
>  include/linux/blkdev.h |  1 +
>  2 files changed, 35 insertions(+)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:07:19 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:07:19 +0200
Subject: [PATCH 04/17] block: add a blk_steal_bios helper
In-Reply-To: <20171023145126.2471-5-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-5-hch@lst.de>
Message-ID: <c1a19718-bb24-0a36-8d89-874c79291bc0@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This helpers allows to bounce steal the uncompleted bios from a request so
> that they can be reissued on another path.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>  block/blk-core.c       | 20 ++++++++++++++++++++
>  include/linux/blkdev.h |  2 ++
>  2 files changed, 22 insertions(+)
> Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:08:21 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:08:21 +0200
Subject: [PATCH 05/17] block: don't look at the struct device dev_t in
 disk_devt
In-Reply-To: <20171023145126.2471-6-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-6-hch@lst.de>
Message-ID: <e4c00ca1-44ca-4e48-cec7-ba5ea15e340e@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> The hidden gendisks introduced in the next patch need to keep the dev
> field in their struct device empty so that udev won't try to create
> block device nodes for them.  To support that rewrite disk_devt to
> look at the major and first_minor fields in the gendisk itself instead
> of looking into the struct device.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  block/genhd.c         | 4 ----
>  include/linux/genhd.h | 2 +-
>  2 files changed, 1 insertion(+), 5 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:18:47 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:18:47 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171023145126.2471-7-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de>
Message-ID: <09f8a87b-f8da-9f5a-6efd-f5e9acfaee45@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> With this flag a driver can create a gendisk that can be used for I/O
> submission inside the kernel, but which is not registered as user
> facing block device.  This will be useful for the NVMe multipath
> implementation.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  block/genhd.c         | 57 +++++++++++++++++++++++++++++++++++----------------
>  include/linux/genhd.h |  1 +
>  2 files changed, 40 insertions(+), 18 deletions(-)
> 
Can we have some information in sysfs to figure out if a gendisk is
hidden? I'd hate having to do an inverse lookup in /proc/partitions;
it's always hard to prove that something is _not_ present.
And we already present various information (like disk_removable_show()),
so it's not without precedent.
And it would make integration with systemd/udev easier.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:20:19 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:20:19 +0200
Subject: [PATCH 07/17] block: add a poll_fn callback to struct
 request_queue
In-Reply-To: <20171023145126.2471-8-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-8-hch@lst.de>
Message-ID: <bffdb2ae-3abf-279f-b07a-ce1f7ea067c4@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> That we we can also poll non blk-mq queues.  Mostly needed for
> the NVMe multipath code, but could also be useful elsewhere.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  block/blk-core.c             | 11 +++++++++++
>  block/blk-mq.c               | 14 +++++---------
>  drivers/nvme/target/io-cmd.c |  2 +-
>  fs/block_dev.c               |  4 ++--
>  fs/direct-io.c               |  2 +-
>  fs/iomap.c                   |  2 +-
>  include/linux/blkdev.h       |  4 +++-
>  mm/page_io.c                 |  2 +-
>  8 files changed, 25 insertions(+), 16 deletions(-)
> Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:21:03 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:21:03 +0200
Subject: [PATCH 08/17] nvme: use kref_get_unless_zero in nvme_find_get_ns
In-Reply-To: <20171023145126.2471-9-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-9-hch@lst.de>
Message-ID: <e6c92822-7c64-b11b-8bb3-41c456ad4c8b@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> For kref_get_unless_zero to protect against lookup vs free races we need
> to use it in all places where we aren't guaranteed to already hold a
> reference.  There is no such guarantee in nvme_find_get_ns, so switch to
> kref_get_unless_zero in this function.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:21:48 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:21:48 +0200
Subject: [PATCH 09/17] nvme: simplify nvme_open
In-Reply-To: <20171023145126.2471-10-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-10-hch@lst.de>
Message-ID: <da101acc-a8a8-0158-df84-4f01bea78a0d@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> Now that we are protected against lookup vs free races for the namespace
> by using kref_get_unless_zero we don't need the hack of NULLing out the
> disk private data during removal.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 40 ++++++++++------------------------------
>  1 file changed, 10 insertions(+), 30 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:23:55 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:23:55 +0200
Subject: [PATCH 10/17] nvme: switch controller refcounting to use struct
 device
In-Reply-To: <20171023145126.2471-11-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-11-hch@lst.de>
Message-ID: <2be01004-39fb-01f9-b0b7-afbe4ee08739@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> Instead of allocating a separate struct device for the character device
> handle embedd it into struct nvme_ctrl and use it for the main controller
> refcounting.  This removes double refcounting and gets us an automatic
> reference for the character device operations.  We keep ctrl->device as a
> pointer for now to avoid chaning printks all over, but in the future we
> could look into message printing helpers that take a controller structure
> similar to what other subsystems do.
> 
> Note the delete_ctrl operation always already has a reference (either
> through sysfs due this change, or because every open file on the
> /dev/nvme-fabrics node has a refernece) when it is entered now, so we
> don't need to do the unless_zero variant there.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  drivers/nvme/host/core.c   | 43 ++++++++++++++++++++++---------------------
>  drivers/nvme/host/fc.c     |  8 ++------
>  drivers/nvme/host/nvme.h   | 12 +++++++++++-
>  drivers/nvme/host/pci.c    |  2 +-
>  drivers/nvme/host/rdma.c   |  5 ++---
>  drivers/nvme/target/loop.c |  2 +-
>  6 files changed, 39 insertions(+), 33 deletions(-)
> Round of applause for this :-)

Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:24:41 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:24:41 +0200
Subject: [PATCH 11/17] nvme: get rid of nvme_ctrl_list
In-Reply-To: <20171023145126.2471-12-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-12-hch@lst.de>
Message-ID: <5b477087-d31d-43a1-2c1e-4d142f4786c3@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> Use the core chrdev code to set up the link between the character device
> and the nvme controller.  This allows us to get rid of the global list
> of all controllers, and also ensures that we have both a reference to
> the controller and the transport module before the open method of the
> character device is called.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sgi at grimberg.me>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 76 ++++++++++--------------------------------------
>  drivers/nvme/host/nvme.h |  3 +-
>  2 files changed, 18 insertions(+), 61 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:25:06 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:25:06 +0200
Subject: [PATCH 12/17] nvme: check for a live controller in nvme_dev_open
In-Reply-To: <20171023145126.2471-13-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-13-hch@lst.de>
Message-ID: <8939bd97-0127-2f15-d243-c110f0e219cf@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This is a much more sensible check than just the admin queue.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at rimbeg.me>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index a56a1e0432e7..df525ab42fcd 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1891,7 +1891,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file)
>  	struct nvme_ctrl *ctrl =
>  		container_of(inode->i_cdev, struct nvme_ctrl, cdev);
>  
> -	if (!ctrl->admin_q)
> +	if (ctrl->state != NVME_CTRL_LIVE)
>  		return -EWOULDBLOCK;
>  	file->private_data = ctrl;
>  	return 0;
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:27:32 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:27:32 +0200
Subject: [PATCH 14/17] nvme: introduce a nvme_ns_ids structure
In-Reply-To: <20171023145126.2471-15-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-15-hch@lst.de>
Message-ID: <2917a1e5-b97d-bc2e-a640-c662a0670b2d@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This allows us to manage the various uniqueue namespace identifiers
> together instead needing various variables and arguments.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Keith Busch <keith.busch at intel.com>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>  drivers/nvme/host/core.c | 69 +++++++++++++++++++++++++++---------------------
>  drivers/nvme/host/nvme.h | 14 +++++++---
>  2 files changed, 49 insertions(+), 34 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From sagi at grimberg.me  Tue Oct 24 00:31:14 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 24 Oct 2017 10:31:14 +0300
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171023145013.GA3017@infradead.org>
References: <20171020161253.29296-1-jsmart2021@gmail.com>
 <20171023082805.GA25383@infradead.org>
 <a1123d71-51f8-1d72-3e5d-67f23634543a@grimberg.me>
 <20171023145013.GA3017@infradead.org>
Message-ID: <b2ec3e9e-2a06-e925-060d-f37e7c7a47cd@grimberg.me>


>> If fatal_err_work calls ->delete_ctrl() and that in turn gets to put the
>> last reference on the ctrl it will end up in ->free_ctrl() under
>> fatal_err_work context which will then try to flush fatal_err_work.
> 
> Yes, and the way I understand flush_work that is perfectly fine for
> flush_work, just not for cancel_work_sync.

Really?

My understanding is that flush_work() is inserting a barrier work
into the worker running the work right after the running work and waits
for it to complete.

How will the barrier complete without the current work completing?

James, can you give it a shot?


From hare at suse.de  Tue Oct 24 00:33:26 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:33:26 +0200
Subject: [PATCH 13/17] nvme: track subsystems
In-Reply-To: <20171023145126.2471-14-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-14-hch@lst.de>
Message-ID: <3670fe01-f17d-92e4-ba51-fdd9fdfd3a6c@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This adds a new nvme_subsystem structure so that we can track multiple
> controllers that belong to a single subsystem.  For now we only use it
> to store the NQN, and to check that we don't have duplicate NQNs unless
> the involved subsystems support multiple controllers.
> 
> Includes code originally from Hannes Reinecke to expose the subsystems
> in sysfs.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  drivers/nvme/host/core.c    | 200 +++++++++++++++++++++++++++++++++++++-------
>  drivers/nvme/host/fabrics.c |   4 +-
>  drivers/nvme/host/nvme.h    |  26 ++++--
>  3 files changed, 194 insertions(+), 36 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:34:11 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:34:11 +0200
Subject: [PATCH 15/17] nvme: track shared namespaces
In-Reply-To: <20171023145126.2471-16-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-16-hch@lst.de>
Message-ID: <bc631b91-a555-5bd7-aa99-61010bcee8b2@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> Introduce a new struct nvme_ns_head that holds information about an actual
> namespace, unlike struct nvme_ns, which only holds the per-controller
> namespace information.  For private namespaces there is a 1:1 relation of
> the two, but for shared namespaces this lets us discover all the paths to
> it.  For now only the identifiers are moved to the new structure, but most
> of the information in struct nvme_ns should eventually move over.
> 
> To allow lockless path lookup the list of nvme_ns structures per
> nvme_ns_head is protected by SRCU, which requires freeing the nvme_ns
> structure through call_srcu.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Keith Busch <keith.busch at intel.com>
> Reviewed-by: Javier Gonz?lez <javier at cnexlabs.com>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c     | 190 +++++++++++++++++++++++++++++++++++++------
>  drivers/nvme/host/lightnvm.c |  14 ++--
>  drivers/nvme/host/nvme.h     |  21 ++++-
>  3 files changed, 190 insertions(+), 35 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:43:11 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:43:11 +0200
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171023145126.2471-17-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
Message-ID: <104278cd-0383-71e0-45c2-750108723b8b@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> This patch adds native multipath support to the nvme driver.  For each
> namespace we create only single block device node, which can be used
> to access that namespace through any of the controllers that refer to it.
> The gendisk for each controllers path to the name space still exists
> inside the kernel, but is hidden from userspace.  The character device
> nodes are still available on a per-controller basis.  A new link from
> the sysfs directory for the subsystem allows to find all controllers
> for a given subsystem.
> 
> Currently we will always send I/O to the first available path, this will
> be changed once the NVMe Asynchronous Namespace Access (ANA) TP is
> ratified and implemented, at which point we will look at the ANA state
> for each namespace.  Another possibility that was prototyped is to
> use the path that is closes to the submitting NUMA code, which will be
> mostly interesting for PCI, but might also be useful for RDMA or FC
> transports in the future.  There is not plan to implement round robin
> or I/O service time path selectors, as those are not scalable with
> the performance rates provided by NVMe.
> 
> The multipath device will go away once all paths to it disappear,
> any delay to keep it alive needs to be implemented at the controller
> level.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  drivers/nvme/host/core.c | 398 ++++++++++++++++++++++++++++++++++++++++-------
>  drivers/nvme/host/nvme.h |  15 +-
>  drivers/nvme/host/pci.c  |   2 +
>  3 files changed, 355 insertions(+), 60 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 1db26729bd89..22c06cd3bef0 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -102,6 +102,20 @@ static int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
>  	return ret;
>  }
>  
> +static void nvme_failover_req(struct request *req)
> +{
> +	struct nvme_ns *ns = req->q->queuedata;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&ns->head->requeue_lock, flags);
> +	blk_steal_bios(&ns->head->requeue_list, req);
> +	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
> +	blk_mq_end_request(req, 0);
> +
> +	nvme_reset_ctrl(ns->ctrl);
> +	kblockd_schedule_work(&ns->head->requeue_work);
> +}
> +
>  static blk_status_t nvme_error_status(struct request *req)
>  {
>  	switch (nvme_req(req)->status & 0x7ff) {
> @@ -129,6 +143,53 @@ static blk_status_t nvme_error_status(struct request *req)
>  	}
>  }
>  
> +static bool nvme_req_needs_failover(struct request *req)
> +{
> +	if (!(req->cmd_flags & REQ_NVME_MPATH))
> +		return false;
> +
> +	switch (nvme_req(req)->status & 0x7ff) {
> +	/*
> +	 * Generic command status:
> +	 */
> +	case NVME_SC_INVALID_OPCODE:
> +	case NVME_SC_INVALID_FIELD:
> +	case NVME_SC_INVALID_NS:
> +	case NVME_SC_LBA_RANGE:
> +	case NVME_SC_CAP_EXCEEDED:
> +	case NVME_SC_RESERVATION_CONFLICT:
> +		return false;
> +
> +	/*
> +	 * I/O command set specific error.  Unfortunately these values are
> +	 * reused for fabrics commands, but those should never get here.
> +	 */
> +	case NVME_SC_BAD_ATTRIBUTES:
> +	case NVME_SC_INVALID_PI:
> +	case NVME_SC_READ_ONLY:
> +	case NVME_SC_ONCS_NOT_SUPPORTED:
> +		WARN_ON_ONCE(nvme_req(req)->cmd->common.opcode ==
> +			nvme_fabrics_command);
> +		return false;
> +
> +	/*
> +	 * Media and Data Integrity Errors:
> +	 */
> +	case NVME_SC_WRITE_FAULT:
> +	case NVME_SC_READ_ERROR:
> +	case NVME_SC_GUARD_CHECK:
> +	case NVME_SC_APPTAG_CHECK:
> +	case NVME_SC_REFTAG_CHECK:
> +	case NVME_SC_COMPARE_FAILED:
> +	case NVME_SC_ACCESS_DENIED:
> +	case NVME_SC_UNWRITTEN_BLOCK:
> +		return false;
> +	}
> +
> +	/* Everything else could be a path failure, so should be retried */
> +	return true;
> +}
> +
>  static inline bool nvme_req_needs_retry(struct request *req)
>  {
>  	if (blk_noretry_request(req))
> @@ -143,6 +204,11 @@ static inline bool nvme_req_needs_retry(struct request *req)
>  void nvme_complete_rq(struct request *req)
>  {
>  	if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
> +		if (nvme_req_needs_failover(req)) {
> +			nvme_failover_req(req);
> +			return;
> +		}
> +
>  		nvme_req(req)->retries++;
>  		blk_mq_requeue_request(req, true);
>  		return;
Sure this works? nvme_req_needs_retry() checks blk_noretry_request():

static inline bool nvme_req_needs_retry(struct request *req)
{
	if (blk_noretry_request(req))
		return false;

which has:
#define blk_noretry_request(rq) \
	((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
			     REQ_FAILFAST_DRIVER))

The original idea here was to _set_ these bits on multipath path devices
so that they won't attempt any retry, but rather forward the I/O error
to the multipath device itself for failover.
So if these bits are set (as they should be for multipathed devices)
we'll never attempt any failover...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Tue Oct 24 00:45:44 2017
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 24 Oct 2017 09:45:44 +0200
Subject: [PATCH 17/17] nvme: also expose the namespace identification
 sysfs files for mpath nodes
In-Reply-To: <20171023145126.2471-18-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-18-hch@lst.de>
Message-ID: <ab2161cc-aa24-4137-8a68-08e63adce764@suse.de>

On 10/23/2017 04:51 PM, Christoph Hellwig wrote:
> We do this by adding a helper that returns the ns_head for a device that
> can belong to either the per-controller or per-subsystem block device
> nodes, and otherwise reuse all the existing code.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Keith Busch <keith.busch at intel.com>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
> ---
>  drivers/nvme/host/core.c | 69 +++++++++++++++++++++++++++++-------------------
>  1 file changed, 42 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 22c06cd3bef0..334735db90c8 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -76,6 +76,7 @@ static DEFINE_IDA(nvme_instance_ida);
>  static dev_t nvme_chr_devt;
>  static struct class *nvme_class;
>  static struct class *nvme_subsys_class;
> +static const struct attribute_group nvme_ns_id_attr_group;
>  
>  static __le32 nvme_get_log_dw10(u8 lid, size_t size)
>  {
> @@ -329,6 +330,8 @@ static void nvme_free_ns_head(struct kref *ref)
>  	struct nvme_ns_head *head =
>  		container_of(ref, struct nvme_ns_head, ref);
>  
> +	sysfs_remove_group(&disk_to_dev(head->disk)->kobj,
> +			   &nvme_ns_id_attr_group);
>  	del_gendisk(head->disk);
>  	blk_set_queue_dying(head->disk->queue);
>  	/* make sure all pending bios are cleaned up */
> @@ -1925,7 +1928,7 @@ static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
>  static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
>  {
>  	struct nvme_subsystem *subsys, *found;
> -	int ret = -ENOMEM;
> +	int ret;
>  
>  	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
>  	if (!subsys)
> @@ -2267,12 +2270,22 @@ static ssize_t nvme_sysfs_rescan(struct device *dev,
>  }
>  static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan);
>  
> +static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
> +{
> +	struct gendisk *disk = dev_to_disk(dev);
> +
> +	if (disk->fops == &nvme_fops)
> +		return nvme_get_ns_from_dev(dev)->head;
> +	else
> +		return disk->private_data;
> +}
> +
>  static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
> -								char *buf)
> +		char *buf)
>  {
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	struct nvme_ns_ids *ids = &ns->head->ids;
> -	struct nvme_subsystem *subsys = ns->ctrl->subsys;
> +	struct nvme_ns_head *head = dev_to_ns_head(dev);
> +	struct nvme_ns_ids *ids = &head->ids;
> +	struct nvme_subsystem *subsys = head->subsys;
>  	int serial_len = sizeof(subsys->serial);
>  	int model_len = sizeof(subsys->model);
>  
> @@ -2294,23 +2307,21 @@ static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
>  
>  	return sprintf(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
>  		serial_len, subsys->serial, model_len, subsys->model,
> -		ns->head->ns_id);
> +		head->ns_id);
>  }
>  static DEVICE_ATTR(wwid, S_IRUGO, wwid_show, NULL);
>  
>  static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
> -			  char *buf)
> +		char *buf)
>  {
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	return sprintf(buf, "%pU\n", ns->head->ids.nguid);
> +	return sprintf(buf, "%pU\n", dev_to_ns_head(dev)->ids.nguid);
>  }
>  static DEVICE_ATTR(nguid, S_IRUGO, nguid_show, NULL);
>  
>  static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
> -								char *buf)
> +		char *buf)
>  {
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	struct nvme_ns_ids *ids = &ns->head->ids;
> +	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
>  
>  	/* For backward compatibility expose the NGUID to userspace if
>  	 * we have no UUID set
> @@ -2325,22 +2336,20 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
>  static DEVICE_ATTR(uuid, S_IRUGO, uuid_show, NULL);
>  
>  static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
> -								char *buf)
> +		char *buf)
>  {
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	return sprintf(buf, "%8phd\n", ns->head->ids.eui64);
> +	return sprintf(buf, "%8phd\n", dev_to_ns_head(dev)->ids.eui64);
>  }
>  static DEVICE_ATTR(eui, S_IRUGO, eui_show, NULL);
>  
>  static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
> -								char *buf)
> +		char *buf)
>  {
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	return sprintf(buf, "%d\n", ns->head->ns_id);
> +	return sprintf(buf, "%d\n", dev_to_ns_head(dev)->ns_id);
>  }
>  static DEVICE_ATTR(nsid, S_IRUGO, nsid_show, NULL);
>  
> -static struct attribute *nvme_ns_attrs[] = {
> +static struct attribute *nvme_ns_id_attrs[] = {
>  	&dev_attr_wwid.attr,
>  	&dev_attr_uuid.attr,
>  	&dev_attr_nguid.attr,
> @@ -2349,12 +2358,11 @@ static struct attribute *nvme_ns_attrs[] = {
>  	NULL,
>  };
>  
> -static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
> +static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
>  		struct attribute *a, int n)
>  {
>  	struct device *dev = container_of(kobj, struct device, kobj);
> -	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
> -	struct nvme_ns_ids *ids = &ns->head->ids;
> +	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
>  
>  	if (a == &dev_attr_uuid.attr) {
>  		if (uuid_is_null(&ids->uuid) ||
> @@ -2372,9 +2380,9 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
>  	return a->mode;
>  }
>  
> -static const struct attribute_group nvme_ns_attr_group = {
> -	.attrs		= nvme_ns_attrs,
> -	.is_visible	= nvme_ns_attrs_are_visible,
> +static const struct attribute_group nvme_ns_id_attr_group = {
> +	.attrs		= nvme_ns_id_attrs,
> +	.is_visible	= nvme_ns_id_attrs_are_visible,
>  };
>  
>  #define nvme_show_str_function(field)						\
> @@ -2883,15 +2891,20 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
>  
>  	device_add_disk(ctrl->device, ns->disk);
>  	if (sysfs_create_group(&disk_to_dev(ns->disk)->kobj,
> -					&nvme_ns_attr_group))
> +					&nvme_ns_id_attr_group))
>  		pr_warn("%s: failed to create sysfs group for identification\n",
>  			ns->disk->disk_name);
>  	if (ns->ndev && nvme_nvm_register_sysfs(ns))
>  		pr_warn("%s: failed to register lightnvm sysfs group for identification\n",
>  			ns->disk->disk_name);
>  
> -	if (new)
> +	if (new) {
>  		device_add_disk(&ns->head->subsys->dev, ns->head->disk);
> +		if (sysfs_create_group(&disk_to_dev(ns->head->disk)->kobj,
> +				&nvme_ns_id_attr_group))
> +			pr_warn("%s: failed to create sysfs group for identification\n",
> +				ns->head->disk->disk_name);
> +	}
>  
>  	return;
>   out_unlink_ns:
device_add_disk_with_groups()?
What happened to that proposal?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From maxg at mellanox.com  Tue Oct 24 01:44:26 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Tue, 24 Oct 2017 11:44:26 +0300
Subject: [PATCH 04/17] block: add a blk_steal_bios helper
In-Reply-To: <20171023145126.2471-5-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-5-hch@lst.de>
Message-ID: <ac15137b-b1a4-3988-17cc-fd6c25749542@mellanox.com>



On 10/23/2017 5:51 PM, Christoph Hellwig wrote:
> This helpers allows to bounce steal the uncompleted bios from a request so
> that they can be reissued on another path.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   block/blk-core.c       | 20 ++++++++++++++++++++
>   include/linux/blkdev.h |  2 ++
>   2 files changed, 22 insertions(+)
> 
> diff --git a/block/blk-core.c b/block/blk-core.c
> index b8c80f39f5fe..e804529e65a5 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -2767,6 +2767,26 @@ struct request *blk_fetch_request(struct request_queue *q)
>   }
>   EXPORT_SYMBOL(blk_fetch_request);
>   
> +/*
> + * Steal bios from a request.  The request must not have been partially
> + * completed before.
> + */

Maybe we can add to the comment that "list" is the destination for the 
stolen bio.

> +void blk_steal_bios(struct bio_list *list, struct request *rq)
> +{
> +	if (rq->bio) {
> +		if (list->tail)
> +			list->tail->bi_next = rq->bio;
> +		else
> +			list->head = rq->bio;
> +		list->tail = rq->biotail;

if list->tail != NULL don't we lose the "list->tail->bi_next = rq->bio;" 
assignment after assigning "list->tail = rq->biotail;" ?

> +	}
> +
> +	rq->bio = NULL;

we can add this NULL assignment inside the big "if", but I'm not sure 
regarding the next 2 assignments.
Anyway not a big deal.

> +	rq->biotail = NULL;
> +	rq->__data_len = 0;
> +}
> +EXPORT_SYMBOL_GPL(blk_steal_bios);
> +



From sagi at grimberg.me  Tue Oct 24 05:25:19 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 24 Oct 2017 15:25:19 +0300
Subject: [PATCH 0/3] centralize transport not ready request check
Message-ID: <1508847922-26699-1-git-send-email-sagi@grimberg.me>

In all fabrics transports, queues are fully functional (live) only
once we are connected in the nvmf level. When we get into reset, delete
or error recovery scenarios with inflight commands, we need to check
that we are ready issue a command.

First check that the queue is LIVE. if not, we can only allow connect
commands to be issued (to help us get to LIVE). Centralize this check
and make all our fabric transport use it.

rdma + loop were tested.

Sagi Grimberg (3):
  nvme-fabrics: Introduce init command check for a queue that is not
    alive
  nvme-fc: Check if queue is ready in queue_rq
  nvme-loop: Check if queue is ready in queue_rq

 drivers/nvme/host/fabrics.h | 30 ++++++++++++++++++++++++++++++
 drivers/nvme/host/fc.c      | 19 ++++++++++++++++++-
 drivers/nvme/host/rdma.c    | 29 ++++++-----------------------
 drivers/nvme/target/loop.c  | 25 ++++++++++++++++++++++++-
 4 files changed, 78 insertions(+), 25 deletions(-)

-- 
2.7.4



From sagi at grimberg.me  Tue Oct 24 05:25:20 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 24 Oct 2017 15:25:20 +0300
Subject: [PATCH 1/3] nvme-fabrics: Introduce init command check for a queue
 that is not alive
In-Reply-To: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508847922-26699-2-git-send-email-sagi@grimberg.me>

When the fabrics queue is not alive and fully functional, no
commands should be allowed to pass but connect (which moves
the queue to a fully functional state). Any other command
should be failed, with either temporary status BLK_STS_RESOUCE
or permanent status BLK_STS_IOERR.

This is shared across all fabrics, hence move the check to
fabrics helper.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.h | 30 ++++++++++++++++++++++++++++++
 drivers/nvme/host/rdma.c    | 29 ++++++-----------------------
 2 files changed, 36 insertions(+), 23 deletions(-)

diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index bf33663218cd..d464cc03cd23 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -142,4 +142,34 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts);
 int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
 bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
 
+static inline blk_status_t nvmf_check_init_req(struct nvme_ctrl *ctrl,
+		struct request *rq)
+{
+	struct nvme_command *cmd = nvme_req(rq)->cmd;
+
+	/*
+	 * We cannot accept any other command until the connect
+	 * command has completed, so only allow connect to pass.
+	 */
+	if (!blk_rq_is_passthrough(rq) ||
+	    cmd->common.opcode != nvme_fabrics_command ||
+	    cmd->fabrics.fctype != nvme_fabrics_type_connect) {
+		/*
+		 * reconnecting state means transport disruption, which
+		 * can take a long time and even might fail permanently,
+		 * fail fast to give upper layers a chance to failover.
+		 * deleting state means that the ctrl will never accept
+		 * commands again, fail it permanently.
+		 */
+		if (ctrl->state == NVME_CTRL_RECONNECTING ||
+		    ctrl->state == NVME_CTRL_DELETING) {
+			nvme_req(rq)->status = NVME_SC_ABORT_REQ;
+			return BLK_STS_IOERR;
+		}
+		return BLK_STS_RESOURCE; /* try again later */
+	}
+
+	return BLK_STS_OK;
+}
+
 #endif /* _NVME_FABRICS_H */
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index a5577e06a620..3e6ece0005e4 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1591,28 +1591,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
  * We cannot accept any other command until the Connect command has completed.
  */
 static inline blk_status_t
-nvme_rdma_queue_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
-{
-	if (unlikely(!test_bit(NVME_RDMA_Q_LIVE, &queue->flags))) {
-		struct nvme_command *cmd = nvme_req(rq)->cmd;
-
-		if (!blk_rq_is_passthrough(rq) ||
-		    cmd->common.opcode != nvme_fabrics_command ||
-		    cmd->fabrics.fctype != nvme_fabrics_type_connect) {
-			/*
-			 * reconnecting state means transport disruption, which
-			 * can take a long time and even might fail permanently,
-			 * so we can't let incoming I/O be requeued forever.
-			 * fail it fast to allow upper layers a chance to
-			 * failover.
-			 */
-			if (queue->ctrl->ctrl.state == NVME_CTRL_RECONNECTING)
-				return BLK_STS_IOERR;
-			return BLK_STS_RESOURCE; /* try again later */
-		}
-	}
-
-	return 0;
+nvme_rdma_is_ready(struct nvme_rdma_queue *queue, struct request *rq)
+{
+	if (unlikely(!test_bit(NVME_RDMA_Q_LIVE, &queue->flags)))
+		return nvmf_check_init_req(&queue->ctrl->ctrl, rq);
+	return BLK_STS_OK;
 }
 
 static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
@@ -1631,7 +1614,7 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 
 	WARN_ON_ONCE(rq->tag < 0);
 
-	ret = nvme_rdma_queue_is_ready(queue, rq);
+	ret = nvme_rdma_is_ready(queue, rq);
 	if (unlikely(ret))
 		return ret;
 
-- 
2.7.4



From sagi at grimberg.me  Tue Oct 24 05:25:21 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 24 Oct 2017 15:25:21 +0300
Subject: [PATCH 2/3] nvme-fc: Check if queue is ready in queue_rq
In-Reply-To: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508847922-26699-3-git-send-email-sagi@grimberg.me>

In case the queue is not LIVE (fully functional
and connected in the nvmf level), we cannot
allow any commands other than connect to pass
through.

Add a new queue state flag NVME_FC_Q_LIVE which is
set after nvmf connect and cleared in queue teardown.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fc.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index c6c903f1b172..7ab131500387 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -41,6 +41,7 @@
 
 enum nvme_fc_queue_flags {
 	NVME_FC_Q_CONNECTED = (1 << 0),
+	NVME_FC_Q_LIVE = (1 << 1),
 };
 
 #define NVMEFC_QUEUE_DELAY	3		/* ms units */
@@ -1715,6 +1716,7 @@ nvme_fc_free_queue(struct nvme_fc_queue *queue)
 	if (!test_and_clear_bit(NVME_FC_Q_CONNECTED, &queue->flags))
 		return;
 
+	clear_bit(NVME_FC_Q_LIVE, &queue->flags);
 	/*
 	 * Current implementation never disconnects a single queue.
 	 * It always terminates a whole association. So there is never
@@ -1722,7 +1724,6 @@ nvme_fc_free_queue(struct nvme_fc_queue *queue)
 	 */
 
 	queue->connection_id = 0;
-	clear_bit(NVME_FC_Q_CONNECTED, &queue->flags);
 }
 
 static void
@@ -1801,6 +1802,8 @@ nvme_fc_connect_io_queues(struct nvme_fc_ctrl *ctrl, u16 qsize)
 		ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
 		if (ret)
 			break;
+
+		set_bit(NVME_FC_Q_LIVE, &ctrl->queues[i].flags);
 	}
 
 	return ret;
@@ -2115,6 +2118,14 @@ nvme_fc_start_fcp_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,
 	return BLK_STS_RESOURCE;
 }
 
+static inline blk_status_t nvme_fc_is_ready(struct nvme_fc_queue *queue,
+		struct request *rq)
+{
+	if (unlikely(!test_bit(NVME_FC_Q_LIVE, &queue->flags)))
+		return nvmf_check_init_req(&queue->ctrl->ctrl, rq);
+	return BLK_STS_OK;
+}
+
 static blk_status_t
 nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx,
 			const struct blk_mq_queue_data *bd)
@@ -2130,6 +2141,10 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx,
 	u32 data_len;
 	blk_status_t ret;
 
+	ret = nvme_fc_is_ready(queue, rq);
+	if (unlikely(ret))
+		return ret;
+
 	ret = nvme_setup_cmd(ns, rq, sqe);
 	if (ret)
 		return ret;
@@ -2465,6 +2480,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	if (ret)
 		goto out_disconnect_admin_queue;
 
+	set_bit(NVME_FC_Q_LIVE, &ctrl->queues[0].flags);
+
 	/*
 	 * Check controller capabilities
 	 *
-- 
2.7.4



From sagi at grimberg.me  Tue Oct 24 05:25:22 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 24 Oct 2017 15:25:22 +0300
Subject: [PATCH 3/3] nvme-loop: Check if queue is ready in queue_rq
In-Reply-To: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
Message-ID: <1508847922-26699-4-git-send-email-sagi@grimberg.me>

In case the queue is not LIVE (fully functional
and connected in the nvmf level), we cannot
allow any commands other than connect to pass
through.

Add a new queue state flag NVME_LOOP_Q_LIVE which is
set after nvmf connect and cleared in queue teardown.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/loop.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c56354e1e4c6..0ddc7a06858a 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -61,10 +61,15 @@ static inline struct nvme_loop_ctrl *to_loop_ctrl(struct nvme_ctrl *ctrl)
 	return container_of(ctrl, struct nvme_loop_ctrl, ctrl);
 }
 
+enum nvme_loop_queue_flags {
+	NVME_LOOP_Q_LIVE	= 0,
+};
+
 struct nvme_loop_queue {
 	struct nvmet_cq		nvme_cq;
 	struct nvmet_sq		nvme_sq;
 	struct nvme_loop_ctrl	*ctrl;
+	unsigned long		flags;
 };
 
 static struct nvmet_port *nvmet_loop_port;
@@ -153,6 +158,14 @@ nvme_loop_timeout(struct request *rq, bool reserved)
 	return BLK_EH_HANDLED;
 }
 
+static inline blk_status_t nvme_loop_is_ready(struct nvme_loop_queue *queue,
+		struct request *rq)
+{
+	if (unlikely(!test_bit(NVME_LOOP_Q_LIVE, &queue->flags)))
+		return nvmf_check_init_req(&queue->ctrl->ctrl, rq);
+	return BLK_STS_OK;
+}
+
 static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx,
 		const struct blk_mq_queue_data *bd)
 {
@@ -162,6 +175,10 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx,
 	struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req);
 	blk_status_t ret;
 
+	ret = nvme_loop_is_ready(queue, req);
+	if (unlikely(ret))
+		return ret;
+
 	ret = nvme_setup_cmd(ns, req, &iod->cmd);
 	if (ret)
 		return ret;
@@ -275,6 +292,7 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_mq_free_tag_set(&ctrl->admin_tag_set);
@@ -305,8 +323,10 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 {
 	int i;
 
-	for (i = 1; i < ctrl->ctrl.queue_count; i++)
+	for (i = 1; i < ctrl->ctrl.queue_count; i++) {
+		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
+	}
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
@@ -346,6 +366,7 @@ static int nvme_loop_connect_io_queues(struct nvme_loop_ctrl *ctrl)
 		ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
 		if (ret)
 			return ret;
+		set_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 	}
 
 	return 0;
@@ -388,6 +409,8 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	if (error)
 		goto out_cleanup_queue;
 
+	set_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+
 	error = nvmf_reg_read64(&ctrl->ctrl, NVME_REG_CAP, &ctrl->ctrl.cap);
 	if (error) {
 		dev_err(ctrl->ctrl.device,
-- 
2.7.4



From rajur at chelsio.com  Tue Oct 24 09:26:51 2017
From: rajur at chelsio.com (Raju  Rangoju)
Date: Tue, 24 Oct 2017 16:26:51 +0000
Subject: kernel paging request error observed on initiator after 'nvmetcli
 clear' on target
Message-ID: <CY1PR12MB07745F65B8684865F1AB8EEABC470@CY1PR12MB0774.namprd12.prod.outlook.com>

Hi Sagi,

I'm seeing an issue with nvmeof, while running fio if I offline CPUs on initiator and 'nvmetcli clear' on target, 'kernel paging request' error is observed on initiator.

This happens at
static int __ib_process_cq(struct ib_cq *cq, int budget)
{
...

    if (wc->wr_cqe)
          wc->wr_cqe->done(cq, wc);
                       ^^^^^^^^^^^

Steps to reproduce:
1. configure target
2. connect to target from client
3. Start fio and offline CPUs on initiator and do 'nvmetcli clear' on target
client side:
fio -filename=/dev/nvme0n1 -iodepth=1 -thread -rw=randwrite -ioengine=psync -bssplit=5k/10:9k/10:13k/10:17k/10:21k/10:25k/10:29k/10:33k/10:37k/10:41k/10 -bs_unaligned -runtime=300 -size=-group_reporting -name=mytest -numjobs=60  &
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 0 > /sys/devices/system/cpu/cpu2/online
echo 0 > /sys/devices/system/cpu/cpu3/online

target side:
nvmetcli clear

-> cqe.wr_id looks like a valid pointer, but it seems the structure that it points to was freed before all the WR completions were reaped.

-> I enabled memory debugging to see if the memory that wr_id was pointing to was freed earlier, but memory debugging did not help.

kernel log:
iw_cxgb4: Chelsio T4/T5 RDMA Driver - version 0.1
iw_cxgb4: 0000:02:00.4: Up
iw_cxgb4: 0000:02:00.4: On-Chip Queues not supported on this device
IPv6: ADDRCONF(NETDEV_UP): enp2s0f4d1: link is not ready
cxgb4 0000:02:00.4 enp2s0f4d1: passive DA module inserted
cxgb4 0000:02:00.4 enp2s0f4d1: link up, 10Gbps, full-duplex, Tx/Rx PAUSE
IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f4d1: link becomes ready
iwpm_register_pid: Unable to send a nlmsg (client = 2)
nvme nvme0: creating 16 I/O queues.
nvme nvme0: new ctrl: NQN "nvme-ram0", addr 102.10.10.238:4420
Broke affinity for irq 120
Broke affinity for irq 125
smpboot: CPU 1 is now offline
IRQ fixup: irq 81 move in progress, old vector 71
Broke affinity for irq 81
Broke affinity for irq 120
Broke affinity for irq 125
smpboot: CPU 2 is now offline
Broke affinity for irq 81
Broke affinity for irq 120
Broke affinity for irq 125
smpboot: CPU 3 is now offline
BUG: unable to handle kernel paging request at 000000000ba30c00
IP: 0xba30c00
PGD 0
P4D 0

Oops: 0010 [#1] SMP DEBUG_PAGEALLOC
CPU: 11 PID: 4180 Comm: kworker/u32:0 Not tainted 4.12.0-rc1 #1
Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
Workqueue: iw_cxgb4 process_work [iw_cxgb4]
task: ffff88084c656780 task.stack: ffffc90001848000
RIP: 0010:0xba30c00
RSP: 0018:ffff88087b6c3ee8 EFLAGS: 00010282
RAX: 0000000000000007 RBX: 00000000000001c0 RCX: ffff8808130d0000
RDX: ffff880812e70168 RSI: ffff880857973400 RDI: ffff880855ae5400
RBP: ffff88087b6c3f20 R08: ffff88087b6c3e90 R09: ffff88087b6c3e8c
R10: ffff880857973d60 R11: ffff8808130e0000 R12: 0000000000000007
R13: 0000000000000080 R14: 0000000000000000 R15: ffff880855ae5400
FS:  0000000000000000(0000) GS:ffff88087b6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000ba30c00 CR3: 0000000001c09000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 ? __ib_process_cq+0x5c/0xb0 [ib_core]
 ib_poll_handler+0x22/0x70 [ib_core]
 irq_poll_softirq+0x98/0xf0
 __do_softirq+0xd0/0x277
 do_softirq_own_stack+0x1c/0x30
 </IRQ>
 do_softirq+0x47/0x50
 __local_bh_enable_ip+0x57/0x60
 t4_ofld_send+0x10d/0x170 [cxgb4]
 cxgb4_remove_tid+0x93/0x110 [cxgb4]
 _c4iw_free_ep+0x58/0x110 [iw_cxgb4]
 close_con_rpl+0x9f/0x180 [iw_cxgb4]
 ? process_work+0x4f/0x60 [iw_cxgb4]
 ? skb_dequeue+0x59/0x70
 process_work+0x43/0x60 [iw_cxgb4]
 process_one_work+0x147/0x370
 worker_thread+0x4a/0x390
 kthread+0x109/0x140
 ? process_one_work+0x370/0x370
 ? kthread_park+0x60/0x60
 ret_from_fork+0x29/0x40
Code:  Bad RIP value.
RIP: 0xba30c00 RSP: ffff88087b6c3ee8
CR2: 000000000ba30c00
---[ end trace 3630f896456a7326 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt
sched: Unexpected reschedule of offline CPU#0!
------------[ cut here ]------------
WARNING: CPU: 11 PID: 4180 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3c/0x40
CPU: 11 PID: 4180 Comm: kworker/u32:0 Tainted: G      D         4.12.0-rc1 #1
Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
Workqueue: iw_cxgb4 process_work [iw_cxgb4]
task: ffff88084c656780 task.stack: ffffc90001848000
RIP: 0010:native_smp_send_reschedule+0x3c/0x40
RSP: 0018:ffff88087b6c3a10 EFLAGS: 00010096
RAX: 000000000000002e RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000001 RSI: ffff88087b6ccba8 RDI: ffff88087b6ccba8
RBP: ffff88087b6c3a10 R08: 0000000000000001 R09: 000000000000081a
R10: 0000000000000059 R11: ffffc9000144fe50 R12: 000000000000000b
R13: 000000010005ec33 R14: ffff88084c656780 R15: ffff88087b6d32a8
FS:  0000000000000000(0000) GS:ffff88087b6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000ba30c00 CR3: 0000000001c09000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 trigger_load_balance+0x10e/0x1f0
 scheduler_tick+0xa1/0xd0
 ? tick_sched_do_timer+0x40/0x40
 update_process_times+0x47/0x60
 tick_sched_handle.isra.15+0x25/0x60
 tick_sched_timer+0x3d/0x70
 __hrtimer_run_queues+0xe3/0x220
 hrtimer_interrupt+0xa8/0x190
 local_apic_timer_interrupt+0x35/0x60
 smp_apic_timer_interrupt+0x38/0x50
 apic_timer_interrupt+0x86/0x90
RIP: 0010:panic+0x1df/0x21d
RSP: 0018:ffff88087b6c3c60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000041 RBX: 0000000000000000 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff88087b6ccba0
RBP: ffff88087b6c3cc8 R08: 0000000000000001 R09: 0000000000000819
R10: 000000000000006c R11: ffffc9000144fe20 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000046
 oops_end+0xb5/0xc0
 no_context+0x17c/0x3d0
 __bad_area_nosemaphore+0xe8/0x1c0
 bad_area_nosemaphore+0x14/0x20
 __do_page_fault+0x89/0x4a0
 ? __radix_tree_lookup+0x75/0xe0
 do_page_fault+0xc/0x10
 page_fault+0x22/0x30
RIP: 0010:0xba30c00
RSP: 0018:ffff88087b6c3ee8 EFLAGS: 00010282
RAX: 0000000000000007 RBX: 00000000000001c0 RCX: ffff8808130d0000
RDX: ffff880812e70168 RSI: ffff880857973400 RDI: ffff880855ae5400
RBP: ffff88087b6c3f20 R08: ffff88087b6c3e90 R09: ffff88087b6c3e8c
R10: ffff880857973d60 R11: ffff8808130e0000 R12: 0000000000000007
R13: 0000000000000080 R14: 0000000000000000 R15: ffff880855ae5400
 ? __ib_process_cq+0x5c/0xb0 [ib_core]
 ? ib_poll_handler+0x22/0x70 [ib_core]
 ? irq_poll_softirq+0x98/0xf0
 ? __do_softirq+0xd0/0x277
 ? do_softirq_own_stack+0x1c/0x30
 </IRQ>
 ? do_softirq+0x47/0x50
 ? __local_bh_enable_ip+0x57/0x60
 ? t4_ofld_send+0x10d/0x170 [cxgb4]
 ? cxgb4_remove_tid+0x93/0x110 [cxgb4]
 ? _c4iw_free_ep+0x58/0x110 [iw_cxgb4]
 ? close_con_rpl+0x9f/0x180 [iw_cxgb4]
 ? process_work+0x4f/0x60 [iw_cxgb4]
 ? skb_dequeue+0x59/0x70
 ? process_work+0x43/0x60 [iw_cxgb4]
 ? process_one_work+0x147/0x370
 ? worker_thread+0x4a/0x390
 ? kthread+0x109/0x140
 ? process_one_work+0x370/0x370
 ? kthread_park+0x60/0x60
 ? ret_from_fork+0x29/0x40
Code: d8 00 0f 92 c0 84 c0 74 14 48 8b 05 6f ba a2 00 be fd 00 00 00 ff 90 a0 00 00 00 5d c3 89 fe 48 c7 c7 58 ee 9c 81 e8 d6 35 11 00 <0f> ff 5d c3 0f 1f 44 00 00 55 be 20 00 08 01 48 89 e5 53 48 89
---[ end trace 3630f896456a7327 ]---

I'm happy to share any further information required.

Thanks,
Raju


From james.smart at broadcom.com  Tue Oct 24 10:55:12 2017
From: james.smart at broadcom.com (James Smart)
Date: Tue, 24 Oct 2017 10:55:12 -0700
Subject: [PATCH v3] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171023220357.11106-1-jsmart2021@gmail.com>
References: <20171023220357.11106-1-jsmart2021@gmail.com>
Message-ID: <650a7f35-0568-8535-926e-2428aa29e237@broadcom.com>

Disregard this version. it does not work. V2 is the valid patch

-- james



From james.smart at broadcom.com  Tue Oct 24 10:55:18 2017
From: james.smart at broadcom.com (James Smart)
Date: Tue, 24 Oct 2017 10:55:18 -0700
Subject: [PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <b2ec3e9e-2a06-e925-060d-f37e7c7a47cd@grimberg.me>
References: <20171020161253.29296-1-jsmart2021@gmail.com>
 <20171023082805.GA25383@infradead.org>
 <a1123d71-51f8-1d72-3e5d-67f23634543a@grimberg.me>
 <20171023145013.GA3017@infradead.org>
 <b2ec3e9e-2a06-e925-060d-f37e7c7a47cd@grimberg.me>
Message-ID: <4ce921ac-501c-c935-05aa-1acce23133c5@broadcom.com>

On 10/24/2017 12:31 AM, Sagi Grimberg wrote:
>
>>> If fatal_err_work calls ->delete_ctrl() and that in turn gets to put 
>>> the
>>> last reference on the ctrl it will end up in ->free_ctrl() under
>>> fatal_err_work context which will then try to flush fatal_err_work.
>>
>> Yes, and the way I understand flush_work that is perfectly fine for
>> flush_work, just not for cancel_work_sync.
>
> Really?
>
> My understanding is that flush_work() is inserting a barrier work
> into the worker running the work right after the running work and waits
> for it to complete.
>
> How will the barrier complete without the current work completing?
>
> James, can you give it a shot?
Sagi is correct.

The patch version that solves this problem is v2:
http://lists.infradead.org/pipermail/linux-nvme/2017-October/013505.html

-- james




From jg at lightnvm.io  Tue Oct 24 10:57:25 2017
From: jg at lightnvm.io (=?utf-8?Q?Javier_Gonz=C3=A1lez?=)
Date: Tue, 24 Oct 2017 19:57:25 +0200
Subject: [PATCH 03/17] block: provide a direct_make_request helper
In-Reply-To: <20171023145126.2471-4-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-4-hch@lst.de>
Message-ID: <6E703CAF-4B4C-47BC-92FE-B6937ECBEF16@lightnvm.io>

> On 23 Oct 2017, at 16.51, Christoph Hellwig <hch at lst.de> wrote:
> 
> This helper allows reinserting a bio into a new queue without much
> overhead, but requires all queue limits to be the same for the upper
> and lower queues, and it does not provide any recursion preventions.
> 
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> ---
> block/blk-core.c       | 34 ++++++++++++++++++++++++++++++++++
> include/linux/blkdev.h |  1 +
> 2 files changed, 35 insertions(+)
> 


Reviewed-by: Javier Gonz?lez <javier at cnexlabs.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20171024/6cf03741/attachment.sig>

From snitzer at redhat.com  Tue Oct 24 14:40:00 2017
From: snitzer at redhat.com (Mike Snitzer)
Date: Tue, 24 Oct 2017 17:40:00 -0400
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171023145126.2471-7-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de>
Message-ID: <20171024213959.GB23307@redhat.com>

On Mon, Oct 23 2017 at 10:51am -0400,
Christoph Hellwig <hch at lst.de> wrote:

> With this flag a driver can create a gendisk that can be used for I/O
> submission inside the kernel, but which is not registered as user
> facing block device.  This will be useful for the NVMe multipath
> implementation.

Having the NVme driver go to such lengths to hide its resources from
upper layers is certainly the work of an evil genius experiencing some
serious territorial issues.  Not sugar-coating it.. you wouldn't.

I kept meaning to reply to your earlier iterations on this series to
ask: can we please get a CONFIG_NVME_MULTIPATHING knob to make it so
that the NVMe driver doesn't implicitly consume (and hide) all
per-controler devices?

Ah well.  There is only one correct way to do NVMe multipathing after
all right?


From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:17 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:17 -0700
Subject: [PATCH V2 00/46] nvmftests: NVMe Over Fabrics Unit Test Framework
Message-ID: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Hi,

This contains second version of the NVMe Over Fabrics unit test framework (nvmftests).

Changes since V1:-
    1. Addition of new testcases to improve the coverage.
    2. Improved directory structure for code clarity.
    3. Libification of the common code.
    4. Support to use different block devices on the target side.
    5. Centralize test configuration management.
    6. Support to execute File System related operations on the host side.
    7. Adjust the code to incorporate the comments received for the
       first version.
    8. Support for setting different log levels for each component.
    9. Support for quick performance measurements.
    10. Class hierarchy documentation and sequence diagram.
    11. Improved workqueue management for host namesapce to
        run parallel commands.

It will be great if I can get some feedback, I'll be happy to incorporate
the comments and make it more usable.

Following is the code repository for current code:-
    https://github.com/ChaitanayaKulkarni/nvmftests/releases

For the purpose of review, this patch series is generated against
$KERN_SRC/tools/testing/selftests/.

Regards,
-Chaitanya

Chaitanya Kulkarni (46):
  nvmftests: add nvmftests README file
  nvmftests-utils: add utility class to define constants
  nvmftests-utils: add shell command package
  nvmftests-utils: add log package
  nvmftests-utils: add diskio package
  nvmftests-utils: add fs package
  nvmftests-utils: add misc package
  nvmftests: add utils package
  nvmftests-nvmf: add target config generator
  nvmftests-nvmf: add support for target ns
  nvmftests-nvmf: add support for target port
  nvmftests-nvmf: add support for target subsystem
  nvmftests-nvmf: add support for target
  nvmftests-nvmf: add target package
  nvmftests-nvmf: add support for host ns
  nvmftests-nvmf: add support for host subsystem
  nvmftests-nvmf: add support for host
  nvmftests-nvmf: add host package
  nvmftests-nvmf: add nvmf core package
  nvmftests-tests: add user config file
  nvmftests-tests: add support for test logger
  nvmftests-tests: add a parent class for tests
  nvmftests-tests: add a test for generic block device
  nvmftests-tests: add a test for ns-descs
  nvmftests-tests: add a test for create/delete host
  nvmftests-tests: add a test for NVMe PCIe target ns
  nvmftests-tests: add a test to create and delete target
  nvmftests-tests: add a test for controller rescan
  nvmftests-tests: add a test for controller reset
  nvmftests-tests: add a test for get-ns-id
  nvmftests-tests: add a test for identify controller
  nvmftests-tests: add a test for identify namespace
  nvmftests-tests: add a test for smart-log
  nvmftests-tests: add a test to run IOs with dd
  nvmftests-tests: add a test to run IOs parallely with dd
  nvmftests-tests: add a test to run mkfs and fio
  nvmftests-tests: add a test to enable/disable target ns
  nvmftests-tests: add a test to run IOs with dd randomly
  nvmftests-tests: add a test to scan host ctrls and ns
  nvmftests-tests: add a test to scan target subsys/ns
  nvmftests-tests: add a target template
  nvmftests-tests: add a host template
  nvmftests-tests: add a test to run traffic and disable ns
  nvmftests-tests: add a test to measure perf with fio
  nvmftests: add class documentation and sequence diagram
  nvmftests: add Makefile

 tools/testing/selftests/nvmftests/Makefile         |  63 +++
 tools/testing/selftests/nvmftests/README.md        | 177 +++++++
 tools/testing/selftests/nvmftests/doc/Makefile     |   7 +
 .../selftests/nvmftests/doc/sequence-diag/Makefile |   7 +
 .../nvmftests/doc/sequence-diag/nvmftests.plantuml |  98 ++++
 tools/testing/selftests/nvmftests/nvmf/__init__.py |  28 ++
 .../selftests/nvmftests/nvmf/host/__init__.py      |  23 +
 .../testing/selftests/nvmftests/nvmf/host/host.py  | 365 ++++++++++++++
 .../selftests/nvmftests/nvmf/host/host_ns.py       | 262 ++++++++++
 .../nvmftests/nvmf/host/host_subsystem.py          | 527 +++++++++++++++++++++
 .../selftests/nvmftests/nvmf/target/__init__.py    |  25 +
 .../selftests/nvmftests/nvmf/target/port.py        | 122 +++++
 .../selftests/nvmftests/nvmf/target/target.py      | 210 ++++++++
 .../nvmf/target/target_config_generator.py         | 222 +++++++++
 .../selftests/nvmftests/nvmf/target/target_ns.py   | 117 +++++
 .../nvmftests/nvmf/target/target_subsystem.py      | 140 ++++++
 .../nvmftests/tests/config/nvmftests.json          |  83 ++++
 .../testing/selftests/nvmftests/tests/nvmf_test.py | 204 ++++++++
 .../selftests/nvmftests/tests/nvmf_test_logger.py  |  50 ++
 .../nvmftests/tests/test_nvmf_create_gen_blk.py    |  71 +++
 .../nvmftests/tests/test_nvmf_create_host.py       |  71 +++
 .../nvmftests/tests/test_nvmf_create_pci_target.py |  70 +++
 .../nvmftests/tests/test_nvmf_create_target.py     |  62 +++
 .../nvmftests/tests/test_nvmf_ctrl_rescan.py       |  63 +++
 .../nvmftests/tests/test_nvmf_ctrl_reset.py        |  63 +++
 .../nvmftests/tests/test_nvmf_get_ns_id.py         |  63 +++
 .../nvmftests/tests/test_nvmf_host_template.py     |  62 +++
 .../selftests/nvmftests/tests/test_nvmf_id_ctrl.py |  63 +++
 .../selftests/nvmftests/tests/test_nvmf_id_ns.py   |  63 +++
 .../selftests/nvmftests/tests/test_nvmf_io.py      |  65 +++
 .../selftests/nvmftests/tests/test_nvmf_mkfs.py    |  65 +++
 .../nvmftests/tests/test_nvmf_ns_descs.py          |  63 +++
 .../nvmftests/tests/test_nvmf_ns_enable_disable.py |  83 ++++
 .../nvmftests/tests/test_nvmf_parallel_io.py       |  65 +++
 .../selftests/nvmftests/tests/test_nvmf_perf.py    |  63 +++
 .../nvmftests/tests/test_nvmf_random_io.py         |  65 +++
 .../nvmftests/tests/test_nvmf_run_host_traffic.py  | 111 +++++
 .../nvmftests/tests/test_nvmf_scan_host.py         |  76 +++
 .../nvmftests/tests/test_nvmf_scan_target.py       |  76 +++
 .../nvmftests/tests/test_nvmf_smart_log.py         |  63 +++
 .../nvmftests/tests/test_nvmf_target_template.py   |  64 +++
 .../testing/selftests/nvmftests/utils/__init__.py  |  32 ++
 .../selftests/nvmftests/utils/const/__init__.py    |  21 +
 .../selftests/nvmftests/utils/const/const.py       |  50 ++
 .../selftests/nvmftests/utils/diskio/__init__.py   |  21 +
 .../testing/selftests/nvmftests/utils/diskio/dd.py |  50 ++
 .../selftests/nvmftests/utils/diskio/fio.py        |  74 +++
 .../selftests/nvmftests/utils/fs/__init__.py       |  21 +
 .../testing/selftests/nvmftests/utils/fs/ext4fs.py | 108 +++++
 .../selftests/nvmftests/utils/fs/filesystem.py     | 122 +++++
 .../selftests/nvmftests/utils/log/__init__.py      |  20 +
 tools/testing/selftests/nvmftests/utils/log/log.py |  58 +++
 .../selftests/nvmftests/utils/misc/__init__.py     |  23 +
 .../nvmftests/utils/misc/generic_blk_dev.py        |  58 +++
 .../selftests/nvmftests/utils/misc/loopback.py     | 103 ++++
 .../selftests/nvmftests/utils/misc/nvme_pci.py     | 114 +++++
 .../selftests/nvmftests/utils/shell/__init__.py    |  20 +
 .../testing/selftests/nvmftests/utils/shell/cmd.py |  47 ++
 58 files changed, 5182 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/Makefile
 create mode 100644 tools/testing/selftests/nvmftests/README.md
 create mode 100644 tools/testing/selftests/nvmftests/doc/Makefile
 create mode 100644 tools/testing/selftests/nvmftests/doc/sequence-diag/Makefile
 create mode 100644 tools/testing/selftests/nvmftests/doc/sequence-diag/nvmftests.plantuml
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host_ns.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host_subsystem.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/port.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_config_generator.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_ns.py
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_subsystem.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/config/nvmftests.json
 create mode 100644 tools/testing/selftests/nvmftests/tests/nvmf_test.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/nvmf_test_logger.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_gen_blk.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_host.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_pci_target.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_target.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_rescan.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_reset.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_get_ns_id.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_host_template.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_id_ctrl.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_id_ns.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_io.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_mkfs.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ns_descs.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ns_enable_disable.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_parallel_io.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_perf.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_random_io.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_run_host_traffic.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_scan_host.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_scan_target.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_smart_log.py
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_target_template.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/const/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/const/const.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/dd.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/fio.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/ext4fs.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/filesystem.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/log/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/log/log.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/generic_blk_dev.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/loopback.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/nvme_pci.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/shell/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/shell/cmd.py

-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:18 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:18 -0700
Subject: [PATCH V2 01/46] nvmftests: add nvmftests README file
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-2-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds readme file for NVMe Over Fabrics unit test
framework (nvmftests).

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 tools/testing/selftests/nvmftests/README.md | 176 ++++++++++++++++++++++++++++
 1 file changed, 177 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/README.md

diff --git a/tools/testing/selftests/nvmftests/README.md b/tools/testing/selftests/nvmftests/README.md
new file mode 100644
index 0000000..b20a96a
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/README.md
@@ -0,0 +1,176 @@
+nvmftests
+=========
+
+
+1. Introduction
+---------------
+
+    This contains NVMe Over Fabrics unit test framework. The purpose of this
+    framework is to provide a platform to create different scenarios and test
+    specific functionality for NVMe Over Fabrics Subsystem (NVMeOF).
+
+2. Overview
+-----------
+
+    The main objective of this framework is to have a platform in place which
+    can provide :-
+    2.1. Classes and methods for each component in the NVMeOF subsystem.
+    2.2. Ability to build NVMeOF subsystem based on the configuration file.
+    2.3. Ability to issue sequential and parallel commands to the different
+         namespaces and controllers from the host and target side.
+
+    All the testcases are written in python3 and nose2 format.
+
+3. Class hierarchy, design considerations and directory structure
+-----------------------------------------------------------------
+
+    This framework follows a simple class hierarchy. Each test is a
+    direct subclass or indirect subclass of NVMeOFTest. To write a new testcase
+    one can copy an existing template test_nvmf_target_template.py or
+    test_nvmf_host_template.py and start adding new testcase specific
+    functionality.
+
+    3.1. Core Classes :- (path $NVMFTESTSHOME/nvmf/)
+       Target Classes :- (path $NVMFTESTSHOME/nvmf/target)
+           NVMeOFTarget :- Represents Target.
+           NVMeOFTargetSubsystem :- Represents a Target Subsystem.
+           NVMeOFTargetNamespace :- Represents a Target Namespace.
+           NVMeOFTargetPort :- Represents a Target Port.
+       Host Classes :- (path $NVMFTESTSHOME/nvmf/host)
+           NVMeOFHost :- Represents Host.
+           NVMeOFHostController :- Represents a Host Controller.
+           NVMeOFHostNamespace :- Represents a Host Namespace.
+
+       We divide host and target side components into two different class
+       hierarchies. On the host side, we have a controller represented as a
+       character device and each namespace as a block device. In order to
+       add any new functionality to the framework please modify core classes
+       for each component and propagate new interfaces to the top
+       level (in host.py/target.py). On the target side, we have subsystem(s),
+       namespace(s), and port(s) which are configured using configfs.
+       For detailed class hierarchy please look into documentation.
+    3.2. Testcase class:-
+        NVMeOFTest :- Base class for each testcase, contains common functions.
+                      Each testcase is direct or indirect subclass of
+                      NVMeOFTest. Testcase class consumes the host and target
+                      classes mentioned in the 3.1 to build the
+                      NVMeOF subsystems.
+    3.3 Directory Structure
+        Following is the quick overview of the directory structure :-
+        .
+        |-- doc                 :- documentation.
+        |   |-- Documentation   :- class documentation.
+        |   |-- sequence-diag   :- sequence diagram.
+        |-- nvmf                :- NVMF core test framework files.
+        |   |-- host            :- NVMF host core files.
+        |   |-- target          :- NVMF target core files.
+        |-- tests               :- test cases.
+        |   |-- config          :- test configuration JSON files.
+        |-- utils               :- utility classes.
+            |-- const           :- constant(s) definitions.
+            |-- diskio          :- diskio related wrappers.
+            |-- fs              :- fs related wrappers.
+            |-- misc            :- miscellaneous files.
+            |-- shell           :- shell command related wrappers.
+            |-- log             :- module logger helpers.
+
+4. Adding new testcases
+-----------------------
+
+    4.1. Please refer to host or target template testcase.
+    4.2. Copy the template file with your testcase name.
+    4.3. Update the class name with testcase name, this has to be unique.
+    4.4. Update the test case function name.
+    4.5. If necessary update the core files and add new functionality.
+    4.6. Add testcase main function to determine success or failure.
+    4.7. Update setUp() and tearDown() to add pre and post functionality.
+    4.8. Once testcase is ready make sure :-
+         4.8.1. Run pep8, flake8, pylint and fix errors/warnings.
+                -Example "$ make static_check" will run pep8, flake8, and
+                pylint on all the python files in current directory.
+         4.8.2. Execute make doc to generate the documentation.
+                -Example "$ make doc" will create and update existing
+                 documentation.
+
+5. Running testcases with framework
+-----------------------------------
+
+    Here are some examples of running testcases with nose2 :-
+        5.1. Running single testcase with nose2 :-
+            from $NVMFTESTSHOME/tests
+            # nose2 --verbose test_nvmf_create_target
+            # nose2 --verbose test_nvmf_create_host
+
+        5.2. Running all the testcases :-
+            from $NVMFTESTSHOME/tests
+            # nose2 --verbose
+
+        5.3 Running all the testcases from makefile :-
+            from #NVMFTESTSHOME
+            # make run
+
+    Some notes on execution:-
+        In the current implementation, it uses file backed loop or nvme-pci
+        block device on the target side. For some testcase execution,
+        a new file is created and linked with loop device. It expects that
+        "mount_path" in the nvmftests.json has enough space available to store
+        backend files which are used for target namespaces. Please edit the
+        target subsystems and namespace configuration, size of the loop device
+        backed file on the target side in the
+        $NVMFTESTSHOME/tests/config/nvmftests.json according to your need.
+
+        For host and target setup, you may have to configure timeout (sleep())
+        values in the code to make sure previous steps are completed
+        successfully and resources are online before executing next the steps.
+        We are planning to make these sleep() calls configurable in the future
+        release.
+
+6. Logging
+----------
+
+    For each testcase, it will create a separate log directory with the test
+    name under logs/. This directory will be used for temporary files and
+    storing execution logs of each testcase. Current implementation stores
+    stdout and stderr for each testcase under log directory, e.g.:-
+        logs/
+        |-- TestNVMFCreateHost
+        |   |-- TestNVMFCreateHost
+        |       |-- stderr.log
+        |       |-- stdout.log
+        |-- TestNVMFCreateTarget
+        |   |?????? TestNVMFCreateTarget
+        |       |-- stderr.log
+        |       |-- stdout.log
+        |-- TestNVMFCtrlRescan
+        |   |-- TestNVMFCtrlRescan
+                .
+                .
+                .
+
+7. Test configuration
+---------------------
+
+    There are two types of config files used by the framework :-
+    7.1. nvmftests.json :- ($NVMFTESTSHOME/tests/config/nvmftests.json)
+         file contains the information about various testcase parameters.
+    7.2. loop.json :- This file is auto-generated by the test framework
+         for the target configuration based on the parameters set in the
+         nvmftests.json.
+
+8. Dependencies
+---------------
+
+    8.1. Python(>= 3.0)
+    8.2. nose(http://nose.readthedocs.io
+    8.3. nose2(Installation guide http://nose2.readthedocs.io/)
+    8.4. pep8(https://pypi.python.org/pypi/setuptools-pep8)
+    8.5. flake8(https://pypi.python.org/pypi/flake8)
+    8.6. pylint(https://www.pylint.org/)
+    8.7. Epydoc(http://epydoc.sourceforge.net/)
+    8.8. nvme-cli(https://github.com/linux-nvme/nvme-cli.git)
+    8.9. fio(https://github.com/axboe/fio)
+
+    Python package management system pip can be used to install most of the
+    listed packages(https://pip.pypa.io/en/stable/installing/) :-
+
+    $ pip install nose nose2 natsort pep8 flake8 pylint epydoc
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:19 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:19 -0700
Subject: [PATCH V2 02/46] nvmftests-utils: add utility class to define
 constants
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-3-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

The new directory utils holds utility classes required
for the framework/tests.

This adds a new class to store all the constants/literals
which are used in the core test framework and tests.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/const/__init__.py    | 21 +++++++++
 .../selftests/nvmftests/utils/const/const.py       | 50 ++++++++++++++++++++++
 2 files changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/const/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/const/const.py

diff --git a/tools/testing/selftests/nvmftests/utils/const/__init__.py b/tools/testing/selftests/nvmftests/utils/const/__init__.py
new file mode 100644
index 0000000..4623af3
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/const/__init__.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+
+from .const import Const
diff --git a/tools/testing/selftests/nvmftests/utils/const/const.py b/tools/testing/selftests/nvmftests/utils/const/const.py
new file mode 100644
index 0000000..128b031
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/const/const.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents General Constants used in the framework.
+"""
+
+
+class Const:
+
+    """
+    Represents a host.
+        - Attributes :
+            General constants for testcases.
+    """
+
+    XXX = "XXX"
+    ONE_KB = 1024
+    ONE_MB = ONE_KB * ONE_KB
+    ONE_GB = ONE_MB * ONE_KB
+
+    KB = "KB"
+    MB = "MB"
+    GB = "GB"
+
+    EXT4 = "ext4"
+
+    CTRL_BLK_FILE_NAME = 2
+    SMART_LOG_VALUE = 1
+    ALLOW_HOST_VALUE = 0
+
+    SYSFS_DEFAULT_MOUNT_PATH = "/sys/kernel/config/"
+    SYSFS_NVMET = "/nvmet/"
+    SYSFS_NVMET_SUBSYS = SYSFS_NVMET + "/subsystems/"
+    SYSFS_NVMET_SUBSYS_NS = "/namespaces/"
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:20 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:20 -0700
Subject: [PATCH V2 03/46] nvmftests-utils: add shell command package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-4-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class which is a wrapper to execute
a shell command and support for the shell package.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/shell/__init__.py    | 20 +++++++++
 .../testing/selftests/nvmftests/utils/shell/cmd.py | 47 ++++++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/shell/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/shell/cmd.py

diff --git a/tools/testing/selftests/nvmftests/utils/shell/__init__.py b/tools/testing/selftests/nvmftests/utils/shell/__init__.py
new file mode 100644
index 0000000..7ddf843
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/shell/__init__.py
@@ -0,0 +1,20 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .cmd import Cmd
diff --git a/tools/testing/selftests/nvmftests/utils/shell/cmd.py b/tools/testing/selftests/nvmftests/utils/shell/cmd.py
new file mode 100644
index 0000000..4a1a645
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/shell/cmd.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents Shell Command execution.
+"""
+import subprocess
+
+
+class Cmd(object):
+
+    """
+    Represents a shell command execution.
+        - Attributes :
+    """
+
+    @staticmethod
+    def exec_cmd(cmd):
+        """ Wrapper for executing a shell command.
+            - Args :
+                - cmd : command to execute.
+            - Returns :
+                - True if cmd returns 0, False otherwise.
+        """
+        proc = None
+        try:
+            proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+        except Exception as err:
+            print(str(err))
+            return False
+
+        return True if proc.wait() == 0 else False
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:21 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:21 -0700
Subject: [PATCH V2 04/46] nvmftests-utils: add log package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-5-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds support to configure logging for the framework.
The new class uses log-level to configure the component
specific logging. Core framework and tests will use
this class to configure the logging.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/log/__init__.py      | 20 ++++++++
 tools/testing/selftests/nvmftests/utils/log/log.py | 58 ++++++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/log/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/log/log.py

diff --git a/tools/testing/selftests/nvmftests/utils/log/__init__.py b/tools/testing/selftests/nvmftests/utils/log/__init__.py
new file mode 100644
index 0000000..adf0b7b
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/log/__init__.py
@@ -0,0 +1,20 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .log import Log
diff --git a/tools/testing/selftests/nvmftests/utils/log/log.py b/tools/testing/selftests/nvmftests/utils/log/log.py
new file mode 100644
index 0000000..cd6ad66
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/log/log.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents Testcaes Logging setup.
+"""
+import json
+import logging
+
+
+class Log(object):
+
+    """
+    Represents a console logger setup helper for a module.
+        - Attributes :
+    """
+
+    @staticmethod
+    def get_logger(name, element):
+        """ Returns the log level from config file.
+            - Args :
+                - element : NVMe Over Fabrics subsystem element.
+            - Returns :
+                - Valid Log level on success, None otherwise.
+        """
+        logger = None
+        with open('config/nvmftests.json') as cfg_file:
+            cfg = json.load(cfg_file)
+            logger = logging.getLogger(name)
+            if cfg['log'][element] == "NOTSET":
+                logger.setLevel(logging.DEBUG)
+            elif cfg['log'][element] == "DEBUG":
+                logger.setLevel(logging.DEBUG)
+            elif cfg['log'][element] == "INFO":
+                logger.setLevel(logging.INFO)
+            elif cfg['log'][element] == "WARNING":
+                logger.setLevel(logging.WARNING)
+            elif cfg['log'][element] == "ERROR":
+                logger.setLevel(logging.ERROR)
+            elif cfg['log'][element] == "CRITICAL":
+                logger.setLevel(logging.CRITICAL)
+
+        return logger
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:22 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:22 -0700
Subject: [PATCH V2 05/46] nvmftests-utils: add diskio package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-6-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds wrapper classes for different I/O utilities.
In current implementation it uses dd(1) and
fio wrapper. These classes will be used by host side tests
to generate I/O workload on host namespace.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/diskio/__init__.py   | 21 ++++++
 .../testing/selftests/nvmftests/utils/diskio/dd.py | 50 +++++++++++++++
 .../selftests/nvmftests/utils/diskio/fio.py        | 74 ++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/dd.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/diskio/fio.py

diff --git a/tools/testing/selftests/nvmftests/utils/diskio/__init__.py b/tools/testing/selftests/nvmftests/utils/diskio/__init__.py
new file mode 100644
index 0000000..52babcf
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/diskio/__init__.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .dd import DD
+from .fio import FIO
diff --git a/tools/testing/selftests/nvmftests/utils/diskio/dd.py b/tools/testing/selftests/nvmftests/utils/diskio/dd.py
new file mode 100644
index 0000000..ad71a8b
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/diskio/dd.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents dd(1) wrapper
+"""
+
+import subprocess
+
+
+class DD(object):
+
+    """
+    Represents dd command wrapper.
+    """
+
+    @staticmethod
+    def run_io(iocfg):
+        """ Executes dd command based on the config argument.
+            - Args :
+                - IO Configuration for dd command.
+            - Returns :
+                - True on success, False on failure.
+        """
+        cmd = "dd if=" + iocfg['IF'] + " of=" + iocfg['OF'] + \
+            " bs=" + iocfg['BS'] + " count=" + iocfg['COUNT'] + \
+            " > /tmp/op 2>&1"
+
+        proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+        ret = True
+        rc = proc.wait()
+        if rc != iocfg['RC']:
+            ret = False
+
+        return ret
diff --git a/tools/testing/selftests/nvmftests/utils/diskio/fio.py b/tools/testing/selftests/nvmftests/utils/diskio/fio.py
new file mode 100644
index 0000000..fbc95e9f
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/diskio/fio.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents FIO wrapper
+"""
+
+import subprocess
+
+
+class FIO(object):
+
+    """
+    Represents fio command wrapper.
+    """
+
+    @staticmethod
+    def run_io(iocfg):
+        """ Executes fio command based on the config argument.
+            - Args :
+                - IO Configuration for fio command.
+            - Returns :
+                - True on success, False on failure.
+        """
+        cmd = "fio "
+        cmd += " --group_reporting=" + iocfg['group_reporting']
+        cmd += " --rw=" + iocfg['rw']
+        cmd += " --bs=" + iocfg['bs']
+        cmd += " --numjobs=" + iocfg['numjobs']
+        cmd += " --iodepth=" + iocfg['iodepth']
+        cmd += " --runtime=" + iocfg['runtime']
+        cmd += " --loops=" + iocfg['loop']
+        cmd += " --ioengine=" + iocfg['ioengine']
+        cmd += " --direct=" + iocfg['direct']
+        cmd += " --invalidate=" + iocfg['invalidate']
+        cmd += " --randrepeat=" + iocfg['randrepeat']
+        cmd += " --time_based "
+        cmd += " --norandommap"
+        cmd += " --exitall"
+        cmd += " --size=" + iocfg['size']
+        if 'filename' in iocfg:
+            cmd += " --filename=" + iocfg['filename']
+            cmd += " --output=" + iocfg['filename'].split('/')[-1] + "_fio.log"
+        else:
+            cmd += " --directory=" + iocfg['directory']
+            cmd += " --output=" + iocfg['directory'].split('/')[-2] + \
+                   "_fio.log"
+        cmd += " --name=" + iocfg['name']
+        ret = True
+        proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+        rc = proc.wait()
+        """
+            some testcases expect fio to fail which is success for the
+            testcase so we test against RC value before deciding success and
+            failure of fio execution
+        """
+        if rc != iocfg['RC']:
+            ret = False
+        return ret
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:25 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:25 -0700
Subject: [PATCH V2 08/46] nvmftests: add utils package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-9-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../testing/selftests/nvmftests/utils/__init__.py  | 32 ++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/__init__.py

diff --git a/tools/testing/selftests/nvmftests/utils/__init__.py b/tools/testing/selftests/nvmftests/utils/__init__.py
new file mode 100644
index 0000000..232b5a5
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/__init__.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .fs import FileSystem
+from .fs import Ext4FS
+
+from .shell import cmd
+
+from .diskio import dd
+from .diskio import fio
+
+from .misc import Loopback
+from .misc import GenBlk
+from .misc import NVMePCIeBlk
+
+from .log import Log
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:26 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:26 -0700
Subject: [PATCH V2 09/46] nvmftests-nvmf: add target config generator
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-10-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to generate target configuration on
the fly. Framework will read the user configuration file and
create a target configuration JSON file on the
fly using this class.

This is useful to have a centralized configuration management.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmf/target/target_config_generator.py         | 222 +++++++++++++++++++++
 1 file changed, 222 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_config_generator.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/target_config_generator.py b/tools/testing/selftests/nvmftests/nvmf/target/target_config_generator.py
new file mode 100644
index 0000000..0da28f9
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/target_config_generator.py
@@ -0,0 +1,222 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Target config file generator.
+"""
+
+import json
+
+
+class Port:
+    """
+    Represents target port config generator.
+
+        - Attributes:
+            - port_id : unique port identification number.
+            - port_dict : dictionary to hold port attributes.
+            - addr : to hold address attributes.
+            - referrals : list of target port referals.
+            - subsystems : list of the subsystems associated with this port.
+    """
+    def __init__(self, port_id):
+        self.port_id = port_id
+        self.port_dict = {}
+        self.addr = {}
+        self.referrals = [None]
+        self.subsystems = []
+
+    def build_addr(self):
+        """ Initialize address attributes for this port.
+            - Args :
+                - None.
+            - Returns :
+                - None.
+        """
+        self.addr['adrfam'] = ""
+        self.addr['traddr'] = ""
+        self.addr['treq'] = "not specified"
+        self.addr['trsvcid'] = ""
+        self.addr['trtype'] = "loop"
+        self.port_dict['addr'] = self.addr
+
+    def build_subsystems(self, subsys_list):
+        """ Initialize subsystem list associated with this port.
+            - Args :
+                - None.
+            -Returns :
+                - Port attributes dictionary.
+        """
+        self.subsystem = subsys_list
+        self.port_dict['subsystems'] = self.subsystem
+
+    def build_port(self, nqn_list):
+        """ Builds port addr, subsystems, id, and referrals.
+            - Args :
+                - List of the subsystem nqn associated with this port.
+            - Returns :
+                - Port dictionary with valid values.
+        """
+        self.build_addr()
+        self.build_subsystems(nqn_list)
+        self.port_dict['portid'] = self.port_id
+        self.port_dict['referrals'] = self.referrals
+        return self.port_dict
+
+
+class Subsystem:
+    """
+    Represents target subsystem config generator.
+
+        - Attributes:
+            - nr_ns : number of namespaces per subsystem.
+            - dev_list : list of block devices.
+            - nqn : subsystem nqn.
+            - ns_list : namespace list for this subsystem.
+            - allowd_hosts : allowd hosts.
+            - attr : subsystem attributes.
+            - namespace : namespace attributes.
+            - device : namespace device attibutes.
+    """
+    def __init__(self, nr_ns, nqn, dev_list):
+        self.nr_ns = nr_ns
+        self.dev_list = dev_list
+        self.nqn = nqn
+        self.ns_list = []
+        self.allowd_hosts = []
+        self.attr = {}
+        self.namespace = {}
+        self.device = {}
+        self.nguid = "00000000-0000-0000-0000-000000000000"
+
+    def add_ns(self, ns_cfg):
+        """ Updates subsystem namespace list with new namespace.
+            - Args :
+                - ns_cfg : namespace configuration.
+            - Returns :
+                - None.
+        """
+        self.allowd_hosts = []
+        self.attr = {}
+        self.namespace = {}
+        self.device = {}
+
+        self.allowd_hosts.append('hostnqn')
+        self.attr['allow_any_host'] = '1'
+        self.device['nguid'] = ns_cfg['device']['nguid']
+        self.device['path'] = ns_cfg['device']['path']
+        self.namespace['device'] = self.device
+        self.namespace['enable'] = ns_cfg['enable']
+        self.namespace['nsid'] = ns_cfg['nsid']
+        ns = self.namespace
+        self.ns_list.append(ns)
+
+    def build_ns(self):
+        """ Build namespace configuration using available devices.
+            - Args :
+                - ns_cfg : namespace configuration.
+            - Returns :
+                - None.
+        """
+        for i in range(0, self.nr_ns):
+            ns_cfg = {}
+            ns_cfg['device'] = {}
+            ns_cfg['device']['nguid'] = self.nguid
+            ns_cfg['device']['path'] = self.dev_list[i % len(self.dev_list)]
+            ns_cfg['enable'] = 1
+            ns_cfg['nsid'] = i + 1
+            self.add_ns(ns_cfg)
+
+    def build_subsys(self):
+        """ Initialize subsystem entry.
+            - Args :
+                - None.
+            - Returns :
+                - subsystem entry dictionary.
+        """
+        self.build_ns()
+        subsys_entry = {}
+        subsys_entry['allowed_hosts'] = self.allowd_hosts
+        subsys_entry['attr'] = self.attr
+        subsys_entry['namespaces'] = self.ns_list
+        subsys_entry['nqn'] = self.nqn
+        subsys = []
+        subsys.append(subsys_entry)
+        return subsys_entry
+
+
+class TargetConfig:
+    """
+    Represents target config generator.
+
+        - Attributes:
+            - subsys_list : list of subsystems associsted with this target.
+            - port_list : list of the ports associated with this target.
+            - config_file_path : path name to test config file.
+            - nr_subsys : number of subsystems present in this target.
+            - nr_ns : number of namespaces per subsystem.
+            - dev_list : list of devices to be used for namespaces.
+    """
+    def __init__(self, config_file_path, nr_subsys, nr_ns, dev_list):
+        self.subsys_list = []
+        self.port_list = []
+        self.config_file_path = config_file_path
+        self.nr_subsys = nr_subsys
+        self.nr_ns = nr_ns
+        self.dev_list = dev_list
+
+    def pp_json(self, json_string, sort=True, indents=4):
+        """ Prints formatted JSON output.
+            - Args :
+                - json_string : JSON string.
+                - sort : flag to sort JSON values before formatting.
+                - idents : identation width.
+            -Returns :
+                - formatted JSON string.
+        """
+        if type(json_string) is str:
+            return json.dumps(json.loads(json_string),
+                              sort_keys=sort, indent=indents)
+        return json.dumps(json_string, sort_keys=sort, indent=indents)
+
+    def build_target_subsys(self):
+        """ Builds the Target config and dumps in JSON format.
+            - Args :
+                - None.
+            - Returns :
+                - None.
+        """
+        nqn_list = []
+        subsys_list = []
+        port_list = []
+        for i in range(0, self.nr_subsys):
+            nqn = "testnqn" + str(i + 1)
+            subsys = Subsystem(self.nr_ns, nqn, self.dev_list)
+            subsys_list.append(subsys.build_subsys())
+            nqn_list.append(nqn)
+
+        # TODO : improve support for multiple ports
+        p = Port(1)
+        port_list.append(p.build_port(nqn_list))
+
+        target_config_str = {}
+        target_config_str['ports'] = port_list
+        target_config_str['subsystems'] = subsys_list
+        data = self.pp_json(target_config_str)
+        with open(self.config_file_path, "w+") as config_file:
+            config_file.write(data)
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:28 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:28 -0700
Subject: [PATCH V2 11/46] nvmftests-nvmf: add support for target port
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-12-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the target port operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/target/port.py        | 122 +++++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/port.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/port.py b/tools/testing/selftests/nvmftests/nvmf/target/port.py
new file mode 100644
index 0000000..a622ea2
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/port.py
@@ -0,0 +1,122 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Target Port.
+"""
+
+import os
+import shutil
+
+from utils.shell import Cmd
+from utils.const import Const
+from utils.log import Log
+
+
+class NVMFTargetPort(object):
+    """
+    Represents a target port.
+
+        - Attributes :
+            - cfgfs : configfs mountpoint.
+            - port_id : port identification number.
+            - port_path : port path in configfs.
+            - port_conf : dictionary to hold the port attributes.
+    """
+    def __init__(self, cfgfs, port_id, **port_conf):
+        self.cfgfs = cfgfs
+        self.port_id = port_id
+        self.port_path = self.cfgfs + "/nvmet/ports/" + port_id + "/"
+        self.port_conf = {}
+        self.port_conf['addr_treq'] = port_conf['addr_treq']
+        self.port_conf['addr_traddr'] = port_conf['addr_traddr']
+        self.port_conf['addr_trtype'] = port_conf['addr_trtype']
+        self.port_conf['addr_adrfam'] = port_conf['addr_adrfam']
+        self.port_conf['addr_trsvcid'] = port_conf['addr_trsvcid']
+        self.port_conf['referrals'] = Const.XXX
+        self.port_conf['subsystems'] = port_conf['subsystems']
+        self.logger = Log.get_logger(__name__, 'port')
+
+    def init(self):
+        """ Create and initialize port.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if self.port_conf['addr_trtype'] != "loop":
+            self.logger.error("only loop transport type is supported.")
+            return False
+
+        ret = Cmd.exec_cmd("mkdir -p " + self.port_path)
+        if ret is False:
+            self.logger.error("failed to create " + self.port_path + ".")
+            return False
+
+        # initialize transport type
+        self.logger.info("Port " + self.port_path + " created successfully.")
+
+        ret = Cmd.exec_cmd("echo -n \"" + self.port_conf['addr_trtype'] +
+                           "\" > " + self.port_path + "/addr_trtype")
+        if ret is False:
+            status = "trtype " + self.port_path + " failed."
+            self.logger.error(status)
+        else:
+            status = "Port " + self.port_path + " initialized successfully."
+            self.logger.info(status)
+        return ret
+
+    def add_subsys(self, subsys_name):
+        """ Link Subsystem to this port.
+            - Args :
+                - subsys_name : subsystem nqn to be linked.
+            - Returns :
+                - True on success, False on failure.
+        """
+        src = self.cfgfs + "/nvmet/subsystems/" + subsys_name
+        if not os.path.exists(src):
+            self.logger.error("subsystem '" + src + "' not present.")
+            return False
+        dest = self.port_path + "/subsystems/"
+        cmd = "ln -s " + src + " " + dest
+        self.logger.info(cmd)
+        return Cmd.exec_cmd(cmd)
+
+    def delete(self):
+        """ Delete this port.
+            - Args :
+                - None.
+            -Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Deleting port " + self.port_id + ".")
+        subsys_symlink = self.port_path + "/subsystem/"
+        try:
+
+            if os.path.isdir(subsys_symlink):
+                shutil.rmtree(subsys_symlink, ignore_errors=True)
+                self.logger.info("Unlink subsystem fromn port successfully.")
+
+            if os.path.isdir(self.port_path):
+                shutil.rmtree(self.port_path, ignore_errors=True)
+
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+        self.logger.info("Removed port " + self.port_path + " successfully.")
+        return True
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:23 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:23 -0700
Subject: [PATCH V2 06/46] nvmftests-utils: add fs package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-7-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds wrapper classes for handling different
file system related operations. In current implementation
it has a support for ext4 file system.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/fs/__init__.py       |  21 ++++
 .../testing/selftests/nvmftests/utils/fs/ext4fs.py | 108 ++++++++++++++++++
 .../selftests/nvmftests/utils/fs/filesystem.py     | 122 +++++++++++++++++++++
 3 files changed, 251 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/ext4fs.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/fs/filesystem.py

diff --git a/tools/testing/selftests/nvmftests/utils/fs/__init__.py b/tools/testing/selftests/nvmftests/utils/fs/__init__.py
new file mode 100644
index 0000000..e83f580
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/fs/__init__.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .filesystem import FileSystem
+from .ext4fs import Ext4FS
diff --git a/tools/testing/selftests/nvmftests/utils/fs/ext4fs.py b/tools/testing/selftests/nvmftests/utils/fs/ext4fs.py
new file mode 100644
index 0000000..e0a5146
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/fs/ext4fs.py
@@ -0,0 +1,108 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents Ext4 File system.
+"""
+import os
+import logging
+
+from .filesystem import FileSystem
+from utils.shell import Cmd
+
+
+class Ext4FS(FileSystem):
+    """
+    Represents Ext4 File System management interface.
+
+        - Attributes :
+    """
+    def __init__(self, dev_path, mount_path=None):
+        super(Ext4FS, self).__init__("ext4", dev_path, mount_path)
+        self.logger = logging.getLogger(__name__)
+        self.log_format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s'
+        self.log_format += '%(filename)20s %(funcName)20s %(lineno)4d'
+        self.log_format += '%(pathname)s'
+        self.formatter = logging.Formatter(self.log_format)
+        self.logger.setLevel(logging.WARNING)
+
+    def get_mount_path(self):
+        """ Accessor for file system mountpath.
+            - Args :
+                - None.
+            - Returns :
+                - File system mountpath.
+        """
+        return super(Ext4FS, self).get_mount_path()
+
+    def mkfs(self):
+        """ Execute mkfs on target deivce.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if super(Ext4FS, self).mkfs() is False:
+            return False
+
+        if Cmd.exec_cmd("mkfs.ext4 " + self.dev_path) is False:
+            self.logger.error("mkfs failed")
+            return False
+
+        self.logger.info("mkfs successful!!!")
+        return True
+
+    def mount(self):
+        """ Mount Target device on the mount path.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if super(Ext4FS, self).mount() is False:
+            return False
+        return Cmd.exec_cmd("mount " + self.dev_path + " " + self.mount_path)
+
+    def is_mounted(self):
+        """ Check if mount_path is mounted.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        return Cmd.exec_cmd("mountpoint -q " + self.mount_path)
+
+    def umount(self):
+        """ Unmount target device from mount path.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if super(Ext4FS, self).umount() is False:
+            return False
+
+        ret = True
+        Cmd.exec_cmd("umount " + self.mount_path)
+        try:
+            os.rmdir(self.mount_path)
+        except Exception as err:
+            self.logger.info(str(err))
+            ret = False
+
+        return ret
diff --git a/tools/testing/selftests/nvmftests/utils/fs/filesystem.py b/tools/testing/selftests/nvmftests/utils/fs/filesystem.py
new file mode 100644
index 0000000..cedb9e9
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/fs/filesystem.py
@@ -0,0 +1,122 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents File System Base class.
+"""
+import os
+import stat
+import logging
+
+
+from utils.shell import Cmd
+
+
+class FileSystem(object):
+    """
+    Represents file system management interface.
+
+        - Attributes :
+            - fs_name : file system name.
+            - dev_path : target device.
+            - mount_path : path to mount target device.
+    """
+    def __init__(self, fs_name, dev_path, mount_path=None):
+        self.fs_name = fs_name
+        self.dev_path = dev_path
+        self.mount_path = mount_path
+        if self.mount_path is None:
+            self.mount_path = "/mnt/" + self.dev_path.split('/')[-1]
+
+        self.logger = logging.getLogger(__name__)
+        self.log_format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s'
+        self.log_format += '%(filename)20s %(funcName)20s %(lineno)4d'
+        self.log_format += '%(pathname)s'
+        self.formatter = logging.Formatter(self.log_format)
+        self.logger.setLevel(logging.WARNING)
+
+    def mkfs(self):
+        """ Check preconditions for the mkfs operation, subclass should
+            implement actual mkfs.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if not os.path.exists(self.dev_path):
+            self.logger.info("ERROR : device path %s is not present.",
+                             self.dev_path)
+            return False
+
+        if not stat.S_ISBLK(os.stat(self.dev_path).st_mode):
+            self.logger.info("ERRO : block device expected for mkfs.")
+            return False
+
+        if self.is_mounted() is True:
+            self.logger.info("ERROR : device is already mounted.")
+            return False
+
+        return True
+
+    def mount(self):
+        """ Check preconditions for the mount operation.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if not os.path.exists(self.mount_path):
+            try:
+                os.makedirs(self.mount_path)
+            except Exception as err:
+                self.logger.info(str(err))
+                return False
+
+        return True
+
+    def get_mount_path(self):
+        """ Accessor for file system mountpath.
+            - Args :
+                - None.
+            - Returns :
+                - File system mountpath.
+        """
+        # TODO : add mount check here
+        return self.mount_path
+
+    def umount(self):
+        """ Check preconditions for the umount operation.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if not self.is_mounted():
+            self.logger.info("fs is not mounted")
+            return False
+
+        return True
+
+    def is_mounted(self):
+        """ Check if mount_path is mounted.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        return Cmd.exec_cmd("mountpoint -q " + self.mount_path)
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:24 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:24 -0700
Subject: [PATCH V2 07/46] nvmftests-utils: add misc package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-8-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds support for misc package. In current implementation
it has different device types which are used on the
NVMe Over Fabrics target namespace. These classes will be
used at the time of target initialization.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/utils/misc/__init__.py     |  23 +++++
 .../nvmftests/utils/misc/generic_blk_dev.py        |  58 +++++++++++
 .../selftests/nvmftests/utils/misc/loopback.py     | 103 +++++++++++++++++++
 .../selftests/nvmftests/utils/misc/nvme_pci.py     | 114 +++++++++++++++++++++
 4 files changed, 298 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/__init__.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/generic_blk_dev.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/loopback.py
 create mode 100644 tools/testing/selftests/nvmftests/utils/misc/nvme_pci.py

diff --git a/tools/testing/selftests/nvmftests/utils/misc/__init__.py b/tools/testing/selftests/nvmftests/utils/misc/__init__.py
new file mode 100644
index 0000000..07806e7
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/misc/__init__.py
@@ -0,0 +1,23 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+
+from .loopback import Loopback
+from .generic_blk_dev import GenBlk
+from .nvme_pci import NVMePCIeBlk
diff --git a/tools/testing/selftests/nvmftests/utils/misc/generic_blk_dev.py b/tools/testing/selftests/nvmftests/utils/misc/generic_blk_dev.py
new file mode 100644
index 0000000..b818cea
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/misc/generic_blk_dev.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents generic block device.
+"""
+
+import os
+import stat
+import logging
+
+from utils.log import Log
+
+
+class GenBlk(object):
+    """
+    Represents Generic BLK block devices.
+
+        - Attributes:
+            - nr_devices : max block devices.
+            - dev_list : list of block devices.
+    """
+    def __init__(self, nr_devices):
+        self.nr_devices = str(nr_devices)
+        self.dev_list = []
+        self.logger = Log.get_logger(__name__, 'gen_blk')
+        self.logger.setLevel(logging.DEBUG)
+
+    def init(self, dev_list):
+        """ Create and initialize Generic Block Device.
+            - Args :
+                - dev_list : list of block devices.
+            - Returns :
+                - True on success, False on failure.
+        """
+        for i in range(0, int(self.nr_devices)):
+            self.logger.info(dev_list[i])
+            if stat.S_ISBLK(os.stat(dev_list[i]).st_mode) is False:
+                self.logger.error("block device not found " + dev_list[i])
+                return False
+            self.dev_list.append(dev_list[i])
+        self.logger.info(self.dev_list)
+        return True
diff --git a/tools/testing/selftests/nvmftests/utils/misc/loopback.py b/tools/testing/selftests/nvmftests/utils/misc/loopback.py
new file mode 100644
index 0000000..04817f5
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/misc/loopback.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents Loopback block devices.
+"""
+
+import os
+import logging
+
+from utils.shell import Cmd
+from utils.log import Log
+
+
+class Loopback(object):
+    """
+    Represents Loopback driver block devices.
+
+        - Attributes:
+            - path : path to create backend files.
+            - dev_size : device file size.
+            - block size : block size to create file.
+            - max_loop : max loop devices.
+            - dev_list : list of loop back files.
+    """
+    def __init__(self, path, dev_size, block_size, max_loop):
+        self.path = path
+        self.dev_size = dev_size
+        self.block_size = block_size
+        self.max_loop = max_loop
+        self.dev_list = []
+        self.logger = Log.get_logger(__name__, 'loopback')
+        self.logger.setLevel(logging.DEBUG)
+
+        Cmd.exec_cmd("losetup -D")
+        Cmd.exec_cmd("modprobe -qr loop")
+        Cmd.exec_cmd("modprobe loop max_loop=" + str(max_loop))
+
+    def init(self):
+        """ Create and initialize Loopback.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if self.dev_size == 0 or self.block_size == 0:
+            self.logger.err("invalid device size or block size")
+            return False
+
+        count = self.dev_size / self.block_size
+
+        for i in range(0, self.max_loop):
+            file_path = self.path + "/test" + str(i)
+            cmd = "dd if=/dev/zero of=" + file_path + \
+                " count=" + str(count) + " bs=" + str(self.block_size)
+            self.logger.info(cmd)
+            ret = Cmd.exec_cmd(cmd)
+            if ret is False:
+                self.logger.error("lookback file creation " + self.dev_list[i])
+                self.delete()
+                return False
+            dev = "/dev/loop" + str(i)
+            cmd = "losetup " + dev + " " + file_path
+            self.logger.info(cmd)
+            if Cmd.exec_cmd(cmd) is False:
+                self.logger.error(cmd + " failed.")
+                return False
+
+            self.dev_list.append(dev)
+        return True
+
+    def delete(self):
+        """ Delete this Loopback.
+            - Args :
+                - None.
+            -Returns :
+                - True on success, False on failure.
+        """
+        loop_cnt = 0
+        for i in self.dev_list:
+            cmd = "losetup -d /dev/loop" + str(loop_cnt)
+            self.logger.info(cmd)
+            Cmd.exec_cmd(cmd)
+            file_path = self.path + "/test" + str(loop_cnt)
+            os.remove(file_path)
+            loop_cnt += 1
+
+        return Cmd.exec_cmd("modprobe -qr loop")
diff --git a/tools/testing/selftests/nvmftests/utils/misc/nvme_pci.py b/tools/testing/selftests/nvmftests/utils/misc/nvme_pci.py
new file mode 100644
index 0000000..f8a4b78
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/utils/misc/nvme_pci.py
@@ -0,0 +1,114 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe PCIe block device.
+"""
+import os
+import re
+import sys
+import time
+from natsort import natsorted
+
+sys.path.append('../../')
+from utils.shell import Cmd
+from utils.log import Log
+
+
+class NVMePCIeBlk(object):
+    """
+    Represents NVMe PCIe block devices.
+
+        - Attributes :
+            - nr_devices : number of block devices.
+            - ctrl_list : NVMe PCIe controller list.
+            - dev_list : list of devices.
+    """
+    def __init__(self):
+        self.nr_devices = None
+        self.ctrl_list = []
+        self.dev_list = []
+        self.logger = Log.get_logger(__name__, 'nvme_pci')
+
+        Cmd.exec_cmd("modprobe -r nvme")
+        Cmd.exec_cmd("modprobe nvme")
+        # allow devices to appear in /dev/
+        time.sleep(1)
+
+    def is_pci_ctrl(self, ctrl):
+        """ Validate underlaying device belongs to pci subsystem.
+            - Args :
+                - None.
+            - Returns :
+                - True if device is belongs to PCIe subsystem, False otherwise.
+        """
+        cmd = "find /sys/devices -name " + ctrl + " | grep -i pci"
+        return Cmd.exec_cmd(cmd)
+
+    def init(self):
+        """ Build the list of NVMe PCIe controllers.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        time.sleep(1)
+        ctrl = "XXX "
+        try:
+            dev_list = os.listdir("/dev/")
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+        dev_list = natsorted(dev_list, key=lambda y: y.lower())
+        pat = re.compile("^nvme[0-9]+$")
+        for line in dev_list:
+            line = line.strip('\n')
+            if pat.match(line):
+                ctrl = line
+                ctrl_dev = "/dev/" + ctrl
+                if self.is_pci_ctrl(ctrl) is True:
+                    self.logger.info("found NVMe PCIe Controller " + ctrl_dev)
+                    self.ctrl_list.append(ctrl_dev)
+                else:
+                    self.logger.info(ctrl_dev + " is not pci ctrl.")
+
+        # find namespace(s) associated with ctrl
+        try:
+            dir_list = os.listdir("/dev/")
+        except Exception, err:
+            self.logger.error(str(err))
+            return False
+        pat = re.compile("^" + ctrl + "+n[0-9]+$")
+        for line in dir_list:
+            line = line.strip('\n')
+            if pat.match(line):
+                self.logger.info("Generated namespace name /dev/" + line + ".")
+                self.dev_list.append("/dev/" + line)
+
+        self.logger.info(self.ctrl_list)
+        self.logger.info(self.dev_list)
+        return True
+
+    def delete(self):
+        """ Remove the nvme module.
+            - Args :
+                - None.
+            -Returns :
+                - True on success, False on failure.
+        """
+        return Cmd.exec_cmd("modprobe -qr nvme")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:27 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:27 -0700
Subject: [PATCH V2 10/46] nvmftests-nvmf: add support for target ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-11-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the target namespace operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/target/target_ns.py   | 117 +++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_ns.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/target_ns.py b/tools/testing/selftests/nvmftests/nvmf/target/target_ns.py
new file mode 100644
index 0000000..0e3ba65
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/target_ns.py
@@ -0,0 +1,117 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Target Namespace.
+"""
+
+import os
+import shutil
+
+from utils.shell import Cmd
+from utils.const import Const
+from utils.log import Log
+
+
+class NVMFTargetNamespace(object):
+    """
+    Represents a target namespace.
+
+        - Attributes :
+            - ns_attr : namespace attributes.
+            - cfgfs : configfs path.
+            - nqn : subsystem nqn.
+            - ns_id : namespace identifier.
+            - ns_path : namespace path in configfs tree.
+            - ns_attr : namespace attributes.
+    """
+    def __init__(self, cfgfs, nqn, ns_id, **ns_attr):
+        self.ns_attr = {}
+        self.cfgfs = cfgfs
+        self.nqn = nqn
+        self.ns_id = ns_id
+        self.ns_path = (self.cfgfs + Const.SYSFS_NVMET_SUBSYS +
+                        nqn + Const.SYSFS_NVMET_SUBSYS_NS + str(ns_id) + "/")
+        self.ns_attr['device_nguid'] = ns_attr['device_nguid']
+        self.ns_attr['device_path'] = ns_attr['device_path']
+        self.ns_attr['enable'] = ns_attr['enable']
+        self.logger = Log.get_logger(__name__, 'target_ns')
+
+    def init(self):
+        """ Create and initialize namespace.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Creating ns " + self.ns_path + " ...")
+        try:
+            os.makedirs(self.ns_path)
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+
+        cmd = "echo -n " + self.ns_attr['device_path'] + " > " + \
+              self.ns_path + "/device_path"
+
+        if Cmd.exec_cmd(cmd) is False:
+            self.logger.error("failed to configure device path.")
+            return False
+
+        if self.ns_attr['enable'] == '1':
+            if self.enable() is False:
+                self.logger.error("enable ns " + self.ns_path + " failed.")
+                return False
+
+        self.logger.info("NS " + self.ns_path + " enabled.")
+        return True
+
+    def disable(self):
+        """ Disable Namespace.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.ns_attr['enable'] = '0'
+        return Cmd.exec_cmd("echo 0 > " + self.ns_path + "/enable")
+
+    def enable(self):
+        """ Enable Namespace.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.ns_attr['enable'] = '1'
+        return Cmd.exec_cmd("echo 1 > " + self.ns_path + "/enable")
+
+    def delete(self):
+        """ Delete namespace.
+            - Args :
+                 - None.
+            - Returns :
+                 - True on success, False on failure.
+        """
+        self.logger.info("Removing NS " + self.ns_path + ".")
+        ret = os.path.exists(self.ns_path)
+        if ret is True:
+            shutil.rmtree(self.ns_path, ignore_errors=True)
+        else:
+            self.logger.error("path " + self.ns_path + " doesn't exists.")
+        return ret
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:29 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:29 -0700
Subject: [PATCH V2 12/46] nvmftests-nvmf: add support for target subsystem
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-13-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the target subsystem operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/nvmf/target/target_subsystem.py      | 140 +++++++++++++++++++++
 1 file changed, 140 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target_subsystem.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/target_subsystem.py b/tools/testing/selftests/nvmftests/nvmf/target/target_subsystem.py
new file mode 100644
index 0000000..abaaecd
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/target_subsystem.py
@@ -0,0 +1,140 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Target Subsystem.
+"""
+
+import os
+import shutil
+
+from utils.shell import Cmd
+from utils.const import Const
+from utils.log import Log
+from nvmf.target.target_ns import NVMFTargetNamespace
+
+
+class NVMFTargetSubsystem(object):
+    """
+    Represents a target subsystem.
+
+        - Attributes :
+            - ns_list : list of namespaces.
+            - cfgfs : configfs path.
+            - nqn : subsystem nqn.
+            - subsys_path : subsystem path in configfs.
+            - allowed_hosts : configfs allowed host attribute.
+            - attr_allow_any_host : configfs allow any host attribute.
+    """
+    def __init__(self, cfgfs, nqn, allowed_hosts, attr_allow_any_host):
+        self.ns_list = []
+        self.cfgfs = cfgfs
+        self.nqn = nqn
+        self.subsys_path = self.cfgfs + Const.SYSFS_NVMET_SUBSYS + nqn + "/"
+        self.allowed_hosts = allowed_hosts
+        self.attr_allow_any_host = attr_allow_any_host
+        self.logger = Log.get_logger(__name__, 'target_subsystem')
+        self.ns_list_index = 0
+
+    def __iter__(self):
+        self.ns_list_index = 0
+        return self
+
+    def __next__(self):
+        index = self.ns_list_index
+        self.ns_list_index += 1
+        if len(self.ns_list) > index:
+            return self.ns_list[index]
+        raise StopIteration
+
+    def next(self):
+        """ Iterator next function """
+        return self.__next__()
+
+    def init(self):
+        """ Create and initialize target subsystem.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        # create subsystem dir
+        self.logger.info("Creating subsys path " + self.subsys_path + ".")
+        try:
+            os.makedirs(self.subsys_path)
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+        # allow any host
+        self.logger.info("Configuring allowed hosts ...")
+        ret = Cmd.exec_cmd("echo " + self.attr_allow_any_host + " >" +
+                           self.subsys_path + "/attr_allow_any_host")
+        if ret is False:
+            self.logger.error(self.subsys_path + " creation failed.")
+        else:
+            self.logger.info(self.subsys_path + " created successfully.")
+        return ret
+
+    def create_ns(self, **ns_attr):
+        """ Create, initialize and store namespace in subsystem's list.
+            - Args :
+                - ns_attr : namespace attributes.
+            - Returns :
+                - namespace handle on success, None on error.
+        """
+        ns_id = len(self.ns_list) + 1
+
+        ns = NVMFTargetNamespace(self.cfgfs, self.nqn, ns_id, **ns_attr)
+        if ns.init() is False:
+            return None
+        self.ns_list.append(ns)
+        return ns
+
+    def delete_ns(self, ns):
+        """ Delete single namespace.
+            - Args :
+                - ns : target namespace object to be deleted.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Deleting namespace " + self.nqn + " : " +
+                         ns.ns_path + ".")
+        ret = ns.delete()
+        if ret is False:
+            self.logger.error("delete ns failed for " + ns.ns_path + ".")
+
+        return ret
+
+    def delete(self):
+        """ Delete subsystem and associated namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Deleting subsystem " + self.nqn)
+        ret = True
+        for ns in iter(self):
+            if self.delete_ns(ns) is False:
+                # try and continue deleting namespaces for cleanup after error
+                ret = False
+
+        if os.path.exists(self.subsys_path):
+            shutil.rmtree(self.subsys_path, ignore_errors=True)
+
+        return ret
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:31 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:31 -0700
Subject: [PATCH V2 14/46] nvmftests-nvmf: add target package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-15-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a support for target package. In current implementation
target package supports port, ns, subsystem and configuration
management in loop mode.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/target/__init__.py    | 25 ++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/__init__.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/__init__.py b/tools/testing/selftests/nvmftests/nvmf/target/__init__.py
new file mode 100644
index 0000000..66a1320
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/__init__.py
@@ -0,0 +1,25 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+
+from .target import NVMFTarget
+from .target_subsystem import NVMFTargetSubsystem
+from .target_ns import NVMFTargetNamespace
+from .port import NVMFTargetPort
+from .target_config_generator import TargetConfig
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:30 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:30 -0700
Subject: [PATCH V2 13/46] nvmftests-nvmf: add support for target
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-14-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the target operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/target/target.py      | 210 +++++++++++++++++++++
 1 file changed, 210 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/target/target.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/target/target.py b/tools/testing/selftests/nvmftests/nvmf/target/target.py
new file mode 100644
index 0000000..87cae14
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/target/target.py
@@ -0,0 +1,210 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Target.
+"""
+
+import sys
+import json
+from nose.tools import assert_equal
+
+from utils.shell import Cmd
+from utils.const import Const
+from utils.log import Log
+from nvmf.target.target_subsystem import NVMFTargetSubsystem
+from nvmf.target.port import NVMFTargetPort
+
+
+class NVMFTarget(object):
+
+    """
+    Represents a target subsystem.
+
+        - Attributes:
+            - subsys_list : list of the subsystem.
+            - port_list : list of the ports.
+            - target_type : target type for ports.
+            - cfgfs : configfs mount point.
+    """
+    def __init__(self, target_type):
+        self.subsys_list = []
+        self.port_list = []
+        self.target_type = target_type
+        self.cfgfs = Const.SYSFS_DEFAULT_MOUNT_PATH
+        self.logger = Log.get_logger(__name__, 'target')
+        self.subsys_list_index = 0
+
+        assert_equal(self.load_configfs(), True)
+
+    def __iter__(self):
+        self.subsys_list_index = 0
+        return self
+
+    def __next__(self):
+        index = self.subsys_list_index
+        self.subsys_list_index += 1
+        if len(self.subsys_list) > index:
+            return self.subsys_list[index]
+        raise StopIteration
+
+    def next(self):
+        """ Iterator next function """
+        return self.__next__()
+
+    def load_configfs(self):
+        """ Load configfs.
+            - Args :
+                - None.
+            -Returns :
+                - True on success, False on failure.
+        """
+        Cmd.exec_cmd("modprobe configfs")
+        ret = Cmd.exec_cmd("mountpoint -q " + self.cfgfs)
+        if ret is False:
+            ret = Cmd.exec_cmd("mount -t configfs none " + self.cfgfs)
+            if ret is False:
+                self.logger.error("failed to mount configfs.")
+                sys.exit(ret)
+            ret = Cmd.exec_cmd("mountpoint -q " + self.cfgfs)
+            if ret is True:
+                self.logger.info("Configfs mounted at " + self.cfgfs + ".")
+            else:
+                self.logger.error("unable to mount configfs at " +
+                                  self.cfgfs + ".")
+        return ret
+
+    def config_loop_target(self, config_file):
+        """ Configure loop target :-
+            1. Create subsystem(s) and respective namespace(s).
+            2. Create port(s) and linked them to respective subsystem(s).
+            3. Create in memory configuration from JSON config file.
+            - Args :
+                - config_file : json config file path.
+            -Returns :
+                - True on success, False on failure.
+        """
+        ret = Cmd.exec_cmd("modprobe nvme-loop")
+        if ret is False:
+            self.logger.error("failed to load nvme-loop.")
+            return False
+
+        try:
+            config_file_handle = open(config_file, "r")
+            config = json.loads(config_file_handle.read())
+            config_file_handle.close()
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+
+        # Subsystem
+        for sscfg in config['subsystems']:
+            # Create Subsystem
+            subsys = NVMFTargetSubsystem(self.cfgfs,
+                                         sscfg['nqn'],
+                                         sscfg['allowed_hosts']
+                                         [Const.ALLOW_HOST_VALUE],
+                                         sscfg['attr']['allow_any_host'])
+            if subsys.init() is False:
+                return False
+
+            self.subsys_list.append(subsys)
+
+            for nscfg in sscfg['namespaces']:
+                ns_attr = {}
+                ns_attr['device_nguid'] = str(nscfg['device']['nguid'])
+                ns_attr['device_path'] = str(nscfg['device']['path'])
+                ns_attr['enable'] = str(nscfg['enable'])
+                ns_attr['nsid'] = str(nscfg['nsid'])
+                if subsys.create_ns(**ns_attr) is False:
+                    return False
+
+        # Port
+        for pcfg in config['ports']:
+            port_cfg = {}
+            port_cfg['addr_treq'] = pcfg['addr']['treq']
+            port_cfg['addr_traddr'] = pcfg['addr']['traddr']
+            port_cfg['addr_trtype'] = pcfg['addr']['trtype']
+            port_cfg['addr_adrfam'] = pcfg['addr']['adrfam']
+            port_cfg['addr_trsvcid'] = pcfg['addr']['trsvcid']
+            port_cfg['portid'] = str(pcfg['portid'])
+            port_cfg['subsystems'] = pcfg['subsystems']
+
+            port = NVMFTargetPort(self.cfgfs, port_cfg['portid'], **port_cfg)
+            if port.init() is False:
+                return False
+
+            self.port_list.append(port)
+
+            for subsys in port_cfg['subsystems']:
+                ret = port.add_subsys(subsys)
+                if ret is False:
+                    self.logger.error("failed to add subsystem " +
+                                      subsys + " to port " +
+                                      port.port_id + ".")
+                    return False
+
+        return True
+
+    def config(self, config_file="config/loop.json"):
+        """ Wrapper for creating target configuration.
+            - Args :
+                - config_file : json config_file.
+            -Returns :
+                - True on success, False on failure.
+        """
+        if Cmd.exec_cmd("modprobe nvme") is False:
+            self.logger.error("failed to load nvme.")
+            return False
+
+        if Cmd.exec_cmd("modprobe nvmet") is False:
+            self.logger.error("unable to load nvmet module.")
+            return False
+
+        ret = False
+        if self.target_type == "loop":
+            self.logger.info("Configuring loop target ... ")
+            ret = self.config_loop_target(config_file)
+        else:
+            self.logger.error("only loop target type is supported.")
+
+        return ret
+
+    def delete(self):
+        """ Target Cleanup.
+            - Args :
+                - None.
+            -Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Cleanup is in progress ...")
+        ret = True
+        for port in self.port_list:
+            if port.delete() is False:
+                ret = False
+
+        for subsys in iter(self):
+            if subsys.delete() is False:
+                ret = False
+
+        self.logger.info("Removing Modules ...")
+        Cmd.exec_cmd("modprobe -r nvme_loop")
+        Cmd.exec_cmd("modprobe -r nvmet")
+        Cmd.exec_cmd("modprobe -r nvme_fabrics")
+        self.logger.info("DONE.")
+        return ret
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:32 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:32 -0700
Subject: [PATCH V2 15/46] nvmftests-nvmf: add support for host ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-16-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the host namespace operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/host/host_ns.py       | 262 +++++++++++++++++++++
 1 file changed, 262 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host_ns.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/host/host_ns.py b/tools/testing/selftests/nvmftests/nvmf/host/host_ns.py
new file mode 100644
index 0000000..42d6f6b
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/host/host_ns.py
@@ -0,0 +1,262 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Host Namespace.
+"""
+
+import Queue
+import time
+import copy
+import threading
+
+from utils.fs import Ext4FS
+from utils.log import Log
+from utils.shell import Cmd
+
+
+class NVMFNSThread(threading.Thread):
+    """
+    Represents a worker thread.
+
+        - Attributes :
+            - target : thread Target.
+            - workq : workqueue shared between producer and worker thread.
+    """
+    def __init__(self, group=None, target=None, name=None,
+                 args=[], kwargs=None, verbose=None):
+        """Default Thread Constructor."""
+        super(NVMFNSThread, self).__init__()
+        self.target = target
+        self.name = name
+        self.workq = args[0]
+        self.q_cond_var = args[1]
+
+    def run(self):
+        """ Default Thread Function """
+        while True:
+            with self.q_cond_var:
+                # if queue is empty wait
+                if self.workq.empty():
+                    self.q_cond_var.wait()
+
+                # get the item from the queue if None exit
+                # use None as signal to exit worker queue
+                item = self.workq.get()
+                if item is None:
+                    break
+
+                ret = item['THREAD'](item)
+                # On Error just shutdown the worker thread
+                # complete all the remaining operations and quit
+                if ret is False:
+                    self.workq.put(None)
+
+
+class NVMFHostNamespace(object):
+    """
+    Represents a host namespace.
+
+        - Attributes :
+            - ns_dev : block device associated with this namespace.
+            - mount_path : mounted directory.
+            - worker_thread : handle for io worker thread.
+            - workq : workqueue shared between producer and worker thread.
+            - q_cond_var : condition variable for queue operations.
+            - fs_type : file system type for mkfs.
+            - fs : file system object.
+    """
+    def __init__(self, ns_dev):
+        self.ns_dev = ns_dev
+        self.mount_path = None
+        self.worker_thread = None
+        self.workq = Queue.Queue()
+        self.q_cond_var = threading.Condition()
+        self.fs_type = None
+        self.fs = None
+        self.logger = Log.get_logger(__name__, 'host_ns')
+
+    def init(self):
+        """ Create worker thread.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if self.id_ns() is False:
+            return False
+
+        # Create worker thread for this ns
+        self.worker_thread = NVMFNSThread(args=[self.workq, self.q_cond_var])
+        self.worker_thread.setDaemon(True)
+        self.worker_thread.start()
+        return True
+
+    def id_ns(self):
+        """ Wrapper for id-ns command.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        id_ns_cmd = "nvme id-ns " + self.ns_dev
+        return Cmd.exec_cmd(id_ns_cmd)
+
+    def ns_descs(self):
+        """ Wrapper for ns-descs command.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ns_descs_cmd = "nvme ns-descs " + self.ns_dev
+        return Cmd.exec_cmd(ns_descs_cmd)
+
+    def get_ns_id(self):
+        """ Wrapper for get-ns-id command.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        get_ns_id_cmd = "nvme get-ns-id " + self.ns_dev
+        return Cmd.exec_cmd(get_ns_id_cmd)
+
+    def mkfs(self, fs_type):
+        """ Format namespace with file system and mount on the unique
+            namespace directory.
+            - Args :
+                - fs_type : file system type.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if fs_type == "ext4":
+            self.fs = Ext4FS(self.ns_dev)
+            if self.fs.mkfs() is True and self.fs.mount() is True:
+                return True
+            else:
+                self.logger.error("mkfs failed for " + self.ns_dev + ".")
+                return False
+        else:
+            self.logger.error("file system is not supported.")
+
+        return False
+
+    def run_fs_ios(self, iocfg):
+        """ Run IOs on mounted file system.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if self.fs.is_mounted() is False:
+            return False
+
+        mount_path = self.fs.get_mount_path()
+        iocfg['directory'] = mount_path + "/"
+
+        if self.worker_thread.is_alive():
+            with self.q_cond_var:
+                self.workq.put(iocfg)
+                self.q_cond_var.notifyAll()
+        else:
+            self.logger.error("worker thread is not running.")
+            return False
+        return True
+
+    def start_io(self, iocfg):
+        """ Add new work IO item to workqueue and notify worker thread.
+            - Args :
+                - IO Configuration passed to worker thread.
+            - Returns :
+                - True on success, False on failure.
+        """
+        iocfg = copy.deepcopy(iocfg)
+        self.logger.info(iocfg)
+        if iocfg['IO_TYPE'] == 'dd':
+            if iocfg['IODIR'] == "read":
+                iocfg['IF'] = self.ns_dev
+            elif iocfg['IODIR'] == "write":
+                iocfg['OF'] = self.ns_dev
+            else:
+                self.logger.error("io config " + iocfg + " not supported.")
+                return False
+        elif iocfg['IO_TYPE'] == 'fio':
+            iocfg['filename'] = self.ns_dev
+        else:
+            self.logger.error("invalid IO type " + iocfg['IO_TYPE'])
+        if self.worker_thread.is_alive():
+            with self.q_cond_var:
+                self.workq.put(iocfg)
+                self.q_cond_var.notifyAll()
+        else:
+            self.logger.error("worker thread is not running.")
+            return False
+
+        return True
+
+    def wait_io(self):
+        """ Wait until queue is empty.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False otherwise.
+        """
+        self.logger.info("Checking for worker thread " + self.ns_dev + ".")
+        if self.worker_thread.is_alive():
+            self.logger.info("Waiting for thread " + self.ns_dev + ".")
+            while not self.workq.empty() and \
+                    self.worker_thread.is_alive() is True:
+                # Wait till waorker thread is alive.
+                time.sleep(1)
+        else:
+            self.logger.error("worker thread is not alive")
+            return False
+        self.logger.info("# WAIT COMPLETE " + self.ns_dev + ".")
+        return True
+
+    def unmount_cleanup(self):
+        """ Unmount the namespace and cleanup the mount path.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if self.fs is not None:
+            return self.fs.umount()
+
+        return True
+
+    def delete(self):
+        """ Namespace clanup.
+            - Args :
+                - None.
+            - Returns :
+                - None.
+        """
+        self.logger.info("delete ns waiting for workq to finish all items")
+        if self.worker_thread.is_alive():
+            with self.q_cond_var:
+                self.workq.put(None)
+                self.q_cond_var.notifyAll()
+
+            while not self.workq.empty() and \
+                    self.worker_thread.is_alive() is True:
+                time.sleep(1)
+
+        self.unmount_cleanup()
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:33 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:33 -0700
Subject: [PATCH V2 16/46] nvmftests-nvmf: add support for host subsystem
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-17-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the host subsystem operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/nvmf/host/host_subsystem.py          | 527 +++++++++++++++++++++
 1 file changed, 527 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host_subsystem.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/host/host_subsystem.py b/tools/testing/selftests/nvmftests/nvmf/host/host_subsystem.py
new file mode 100644
index 0000000..e32d952
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/host/host_subsystem.py
@@ -0,0 +1,527 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Host Controller.
+"""
+
+import re
+import os
+import stat
+import time
+import random
+import string
+import subprocess
+from natsort import natsorted
+
+from utils.const import Const
+from utils.shell import Cmd
+from utils.log import Log
+from nvmf.host.host_ns import NVMFHostNamespace
+
+
+class NVMFHostController(object):
+    """
+    Represents a host controller.
+
+        - Attributes :
+            - nqn : ctrl nqn.
+            - ctrl_dev : controller device.
+            - ctrl_dict : controller attributes.
+            - ns_list : list of namespaces.
+            - ns_dev_list : namespace device list.
+            - transport : transport type.
+    """
+    def __init__(self, nqn, transport):
+        self.nqn = nqn
+        self.ctrl_dev = None
+        self.ctrl_dict = {}
+        self.ns_list = []
+        self.ns_dev_list = []
+        self.transport = transport
+        self.ns_list_index = 0
+        self.logger = Log.get_logger(__name__, 'host_subsystem')
+
+    def __iter__(self):
+        self.ns_list_index = 0
+        return self
+
+    def __next__(self):
+        index = self.ns_list_index
+        self.ns_list_index += 1
+        if (len(self.ns_list) > index):
+            return self.ns_list[index]
+        raise StopIteration
+
+    def next(self):
+        """ Iterator next function """
+        return self.__next__()
+
+    def run_io_all_ns(self, iocfg):
+        """ Start IOs on all the namespaces of this controller parallelly.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ns in iter(self):
+            try:
+                if ns.start_io(iocfg) is False:
+                    self.logger.error("start IO " + ns.ns_dev + ".")
+                    ret = False
+                    break
+                self.logger.info("start IO " + ns.ns_dev + " SUCCESS.")
+            except StopIteration:
+                break
+
+        return ret
+
+    def wait_io_all_ns(self):
+        """ Wait until workqueue is empty.
+            - Args :
+                - None.
+            - Returns :
+                - True if namespace wait IO is successful, False otherwise.
+        """
+        for ns in iter(self):
+            try:
+                ret = ns.wait_io()
+                if ret is False:
+                    return False
+            except StopIteration:
+                break
+        return True
+
+    def run_io_seq(self, iocfg):
+        """ Exercise IOs on each namespace.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ns in iter(self):
+            try:
+                if ns.start_io(iocfg) is False:
+                    self.logger.error("start IO " + ns.ns_dev + ".")
+                    ret = False
+                    break
+                ret = ns.wait_io()
+            except StopIteration:
+                break
+        return ret
+
+    def run_mkfs_seq(self, fs_type):
+        """ Run mkfs.
+            - Args :
+                - fs_type : file system type.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ns in iter(self):
+            try:
+                if ns.mkfs(fs_type) is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def run_fs_ios(self, iocfg):
+        """ Run IOs on mounted file system.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ns in iter(self):
+            try:
+                if ns.run_fs_ios(iocfg) is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def run_io_random(self, iocfg):
+        """ Select the namespce randomely and wait for the IOs completion,
+            repeat this for all the namespaces.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        num_list = range(0, len(self.ns_list))
+
+        for i in range(0, len(self.ns_list)):
+            random.shuffle(num_list)
+            ns_id = num_list.pop()
+
+            host_ns = self.ns_list[ns_id]
+            if host_ns.start_io(iocfg) is False:
+                return False
+            host_ns.wait_io()
+
+        return True
+
+    def __ctrl_set_attr__(self, attr):
+        """ Set host controller attribute.
+            - Args :
+                - attr : sysfs attribute to set.
+            - Returns :
+                - True on success, False on failure.
+        """
+        sysfs_path = "/sys/class/nvme-fabrics/ctl/"
+        cmd = "echo 1 >" + sysfs_path + self.ctrl_dev.split('/')[-1] \
+              + "/" + attr
+        return Cmd.exec_cmd(cmd)
+
+    def ctrl_rescan(self):
+        """ Issue controller rescan.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        return self.__ctrl_set_attr__("rescan_controller")
+
+    def ctrl_reset(self):
+        """ Issue controller reset.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        return self.__ctrl_set_attr__("reset_controller")
+
+    def run_smart_log(self, nsid="0xFFFFFFFF"):
+        """ Wrapper for nvme smart-log command.
+            - Args :
+                - nsid : namespace id for smart log, defaults to ctrl.
+            - Returns:
+                - True on success, False on failure.
+        """
+        smart_log_cmd = "nvme smart-log " + self.ctrl_dev + " -n " + str(nsid)
+        self.logger.info(smart_log_cmd)
+        proc = subprocess.Popen(smart_log_cmd,
+                                shell=True,
+                                stdout=subprocess.PIPE)
+        err = proc.wait()
+        if err != 0:
+            self.logger.error("nvme smart log failed.")
+            return False
+
+        for line in proc.stdout:
+            if "data_units_read" in line:
+                temp_str = line.split(":")[Const.SMART_LOG_VALUE].strip()
+                data_units_read = string.replace(temp_str, ",", "")
+            if "data_units_written" in line:
+                temp_str = line.split(":")[Const.SMART_LOG_VALUE].strip()
+                data_units_written = string.replace(temp_str, ",", "")
+            if "host_read_commands" in line:
+                temp_str = line.split(":")[Const.SMART_LOG_VALUE].strip()
+                host_read_commands = string.replace(temp_str, ",", "")
+            if "host_write_commands" in line:
+                temp_str = line.split(":")[Const.SMART_LOG_VALUE].strip()
+                host_write_commands = string.replace(temp_str, ",", "")
+
+        self.logger.info("data_units_read " + data_units_read)
+        self.logger.info("data_units_written " + data_units_written)
+        self.logger.info("host_read_commands " + host_read_commands)
+        self.logger.info("host_write_commands " + host_write_commands)
+        return True
+
+    def smart_log(self):
+        """ Execute smart-log command.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        # Run smart log on controller
+        self.run_smart_log()
+        i = 1
+        for ns in iter(self):
+            try:
+                if self.run_smart_log(i) is False:
+                    return False
+                i += 1
+            except StopIteration:
+                break
+        return True
+
+    def validate_sysfs_ns(self):
+        """ Validate sysfs entries for the host controller and namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ctrl_bdev = self.ctrl_dev.split("/")[Const.CTRL_BLK_FILE_NAME]
+        # Validate ctrl in the sysfs
+        cmd = "basename $(dirname $(grep -ls " + self.nqn + \
+              " /sys/class/nvme-fabrics/ctl/*/subsysnqn))"
+        proc = subprocess.Popen(cmd,
+                                shell=True,
+                                stdout=subprocess.PIPE)
+        for line in proc.stdout:
+            line = line.strip('\n')
+            # compare nvmeN in /dev/nvmeN in sysfs
+            if line != ctrl_bdev:
+                self.logger.error("host ctrl " + self.ctrl_dev +
+                                  " not present.")
+                return False
+        dir_list = os.listdir("/sys/class/nvme-fabrics/ctl/" + ctrl_bdev + "/")
+
+        pat = re.compile("^" + ctrl_bdev + "+n[0-9]+$")
+        for line in dir_list:
+            line = line.strip('\n')
+            if pat.match(line):
+                if "/dev/" + line not in self.ns_dev_list:
+                    self.logger.error("ns " + line + " not found in sysfs.")
+                    return False
+
+        self.logger.info("sysfs entries for ctrl and ns created successfully.")
+        return True
+
+    def build_ns_list(self):
+        """ Generate next available controller and namespace id on the fly.
+            Build the ns list for this controller.
+            - Args :
+                - None.
+            - Returns :
+                - ctrl and ns list on success, None on failure.
+        """
+        ctrl = Const.XXX
+        ns_list = []
+        try:
+            dev_list = os.listdir("/dev/")
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return None, None
+        dev_list = natsorted(dev_list, key=lambda y: y.lower())
+        # we assume that atleast one namespace is created on target
+        # find ctrl
+        pat = re.compile("^nvme[0-9]+$")
+        for line in dev_list:
+            line = line.strip('\n')
+            if pat.match(line):
+                ctrl = line
+
+        if ctrl == Const.XXX:
+            self.logger.error("controller '/dev/nvme*' not found.")
+            return None, None
+        # allow namespaces to appear in the /dev/
+        time.sleep(2)
+        # find namespace(s) associated with ctrl
+        try:
+            dir_list = os.listdir("/dev/")
+        except Exception, err:
+            self.logger.error(str(err))
+            return None, None
+        pat = re.compile("^" + ctrl + "+n[0-9]+$")
+        for line in dir_list:
+            line = line.strip('\n')
+            if pat.match(line):
+                self.logger.info("Generated namespace name /dev/" + line + ".")
+                ns_list.append("/dev/" + line)
+
+        if len(ns_list) == 0:
+            self.logger.error("host ns not found for ctrl " + ctrl + ".")
+            return None, None
+
+        ctrl = "/dev/" + ctrl
+        return ctrl, ns_list
+
+    def init_ns(self):
+        """ Initialize and build namespace list and validate sysfs entries.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Expecting following namespaces " +
+                         str(self.ns_dev_list) + ".")
+        for ns_dev in self.ns_dev_list:
+            if not stat.S_ISBLK(os.stat(ns_dev).st_mode):
+                self.logger.error("expected block dev " + ns_dev + ".")
+                return False
+
+            self.logger.info("Found NS " + ns_dev + ".")
+            host_ns = NVMFHostNamespace(ns_dev)
+            host_ns.init()
+            self.ns_list.append(host_ns)
+        # allow sysfs entries to populate
+        time.sleep(1)
+        if self.validate_sysfs_ns() is False:
+            self.logger.error("unable to verify sysfs entries.")
+            return False
+        self.logger.info("Host sysfs entries are validated.")
+        return True
+
+    def validate_fabric_ctrl(self):
+        """ Validate this is a fabric controller.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+
+        if not stat.S_ISCHR(os.stat(self.ctrl_dev).st_mode):
+            self.logger.error("failed to find char device for host ctrl.")
+
+        cmd = "find /sys/devices -name " + self.ctrl_dev.split("/")[-2] \
+              + " | grep nvme-fabric"
+        self.logger.info(cmd)
+        return Cmd.exec_cmd(cmd)
+
+    def init_ctrl(self):
+        """ Initialize controller and build controller attributes.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        # initialize nqn and transport
+        cmd = "echo  \"transport=" + self.transport + ",nqn=" + \
+              self.nqn + "\" > /dev/nvme-fabrics"
+        self.logger.info("Host Connect command : " + cmd)
+        if Cmd.exec_cmd(cmd) is False:
+            self.logger.info("ERROR : host connect command failed")
+            return False
+        self.ctrl_dev, self.ns_dev_list = self.build_ns_list()
+
+        if self.validate_fabric_ctrl() is False or self.id_ctrl() is False:
+            return False
+
+        return self.init_ns()
+
+    def id_ctrl(self):
+        """ Wrapper for executing id-ctrl command.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        id_ctrl_cmd = "nvme id-ctrl " + self.ctrl_dev
+        proc = subprocess.Popen(id_ctrl_cmd,
+                                shell=True,
+                                stdout=subprocess.PIPE)
+        err = proc.wait()
+        if err != 0:
+            self.logger.error("nvme id-ctrl failed.")
+            return False
+
+        for line in proc.stdout:
+            if line.startswith('subnqn') or \
+               line.startswith('NVME Identify Controller'):
+                continue
+            if line.startswith('ps ') or line.strip().startswith('rwt'):
+                continue
+            key, val = line.split(':')
+            self.ctrl_dict[key.strip()] = val.strip()
+
+        self.logger.info("ID controller :- ")
+        self.logger.info(self.ctrl_dict)
+        return True
+
+    def id_ns(self):
+        """ Wrapper for executing id-ns command on all namespaces of this
+            controller.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        for host_ns in iter(self):
+            if host_ns.id_ns() is False:
+                return False
+        return True
+
+    def ns_descs(self):
+        """ Wrapper for executing ns_descs command on all namespaces of this
+            controller.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        for host_ns in iter(self):
+            if host_ns.ns_descs() is False:
+                return False
+        return True
+
+    def get_ns_id(self):
+        """ Wrapper for executing get-ns-id command on all namespaces of this
+            controller.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        for host_ns in iter(self):
+            if host_ns.get_ns_id() is False:
+                return False
+        return True
+
+    def generate_next_ns_id(self):
+        """ Return next namespace id.
+            - Args :
+                - None.
+            - Returns :
+                - next namespace id.
+        """
+        return len(self.ns_list) + 1
+
+    def delete(self):
+        """ Delete subsystem and associated namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Deleting subsystem " + self.nqn + ".")
+        for host_ns in self.ns_list:
+            host_ns.delete()
+        cmd = "dirname $(grep -ls " + self.nqn + \
+              " /sys/class/nvme-fabrics/ctl/*/subsysnqn)"
+        try:
+            proc = subprocess.Popen(cmd,
+                                    shell=True,
+                                    stdout=subprocess.PIPE)
+            for line in proc.stdout:
+                line = line.strip('\n')
+                if not os.path.isdir(line):
+                    self.logger.error("host ctrl dir " + self.nqn +
+                                      " not present.")
+                    return False
+                cmd = "nvme disconnect -n " + self.nqn
+                self.logger.info("disconnecting : " + cmd)
+                ret = Cmd.exec_cmd(cmd)
+                if ret is False:
+                    self.logger.error("failed to delete ctrl " +
+                                      self.nqn + ".")
+                    return False
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+
+        return True
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:35 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:35 -0700
Subject: [PATCH V2 18/46] nvmftests-nvmf: add host package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-19-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a support for host package. In current implementation
host package supports ns, subsystem and configuration
management in loop mode.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/nvmf/host/__init__.py      | 23 ++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/__init__.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/host/__init__.py b/tools/testing/selftests/nvmftests/nvmf/host/__init__.py
new file mode 100644
index 0000000..10a2e4ee
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/host/__init__.py
@@ -0,0 +1,23 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+
+from .host import NVMFHost
+from .host_subsystem import NVMFHostController
+from .host_ns import NVMFHostNamespace
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:36 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:36 -0700
Subject: [PATCH V2 19/46] nvmftests-nvmf: add nvmf core package
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-20-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 tools/testing/selftests/nvmftests/nvmf/__init__.py | 28 ++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/__init__.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/__init__.py b/tools/testing/selftests/nvmftests/nvmf/__init__.py
new file mode 100644
index 0000000..57c3985
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/__init__.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+from .host import NVMFHost
+from .host import NVMFHostController
+from .host import NVMFHostNamespace
+
+from .target import NVMFTarget
+from .target import NVMFTargetSubsystem
+from .target import NVMFTargetNamespace
+from .target import NVMFTargetPort
+from .target import TargetConfig
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:34 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:34 -0700
Subject: [PATCH V2 17/46] nvmftests-nvmf: add support for host
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-18-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new class to handle the host operations.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../testing/selftests/nvmftests/nvmf/host/host.py  | 365 +++++++++++++++++++++
 1 file changed, 365 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/nvmf/host/host.py

diff --git a/tools/testing/selftests/nvmftests/nvmf/host/host.py b/tools/testing/selftests/nvmftests/nvmf/host/host.py
new file mode 100644
index 0000000..d14fb77
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/nvmf/host/host.py
@@ -0,0 +1,365 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Represents NVMe Over Fabric Host Subsystem.
+"""
+
+import json
+import random
+from nose.tools import assert_equal
+
+from utils.shell import Cmd
+from utils.log import Log
+from nvmf.host.host_subsystem import NVMFHostController
+
+
+class NVMFHost(object):
+
+    """
+    Represents a host.
+
+        - Attributes :
+              - target_type : rdma/loop/fc. (only loop supported now)
+              - ctrl_list : list of the host controllers.
+    """
+    def __init__(self, target_type):
+        self.target_type = target_type
+        self.ctrl_list = []
+        self.ctrl_list_index = 0
+        self.logger = Log.get_logger(__name__, 'host')
+        assert_equal(self.load_modules(), True)
+
+    def __iter__(self):
+        self.ctrl_list_index = 0
+        return self
+
+    def __next__(self):
+        index = self.ctrl_list_index
+        self.ctrl_list_index += 1
+        if len(self.ctrl_list) > index:
+            return self.ctrl_list[index]
+        raise StopIteration
+
+    def next(self):
+        """ Iterator next function """
+        return self.__next__()
+
+    def load_modules(self):
+        """ Wrapper for Loading NVMF Host modules.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        if Cmd.exec_cmd("modprobe nvme-fabrics") is False:
+            self.logger.error("unable to load nvme-fabrics.")
+            return False
+        return True
+
+    def config_loop(self, config_file):
+        """ Configure host for loop target :-
+            1. Load config from json file.
+            2. Create Controller list.
+            - Args :
+                - config_file : json config file.
+            -Returns :
+                - True on success, False on failure.
+        """
+        try:
+            config_file_handle = open(config_file, "r")
+            config = json.loads(config_file_handle.read())
+            config_file_handle.close()
+        except Exception, err:
+            self.logger.error(str(err) + ".")
+            return False
+
+        for sscfg in config['subsystems']:
+            ctrl = NVMFHostController(sscfg['nqn'], "loop")
+            if ctrl.init_ctrl()is False:
+                self.logger.error("ctrl init " + str(ctrl.ctrl_dev) + ".")
+                return False
+            self.ctrl_list.append(ctrl)
+        return True
+
+    def run_traffic_parallel(self, iocfg):
+        """ Run parallel IO traffic on all host controller(s) and
+            wait for completion.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - None.
+        """
+        ret = True
+        self.logger.info("Starting traffic parallelly on all controllers ...")
+        for ctrl in iter(self):
+            if ctrl.run_io_all_ns(iocfg) is False:
+                ret = False
+        return ret
+
+    def wait_traffic_parallel(self):
+        """ Wait for IOs completion on all controllers.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - None.
+        """
+        ret = True
+        self.logger.info("Waiting for all threads to finish the IOs ...")
+        for ctrl in iter(self):
+            if ctrl.wait_io_all_ns() is False:
+                self.logger.error("wait on " + ctrl.ctrl_dev + ".")
+                ret = False
+
+        return ret
+
+    def run_ios_parallel(self, iocfg):
+        """ Run parallel IOs on all host controller(s) and
+            wait for completion.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - None.
+        """
+        if self.run_traffic_parallel(iocfg) is False:
+            return False
+
+        if self.wait_traffic_parallel() is False:
+            return False
+
+        return True
+
+    def run_perf_parallel(self, iocfg):
+        """ Run parallel IO traffic on all host controller(s) and
+            wait for completion.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - None.
+        """
+        if self.run_ios_parallel(iocfg) is False:
+            return False
+
+        # add perf report generation mechanism here
+
+        return True
+
+    def run_ios_seq(self, iocfg):
+        """ Run IOs on all host controllers sequentially.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        self.logger.info("Starting IOs seq ...")
+        ret = True
+        for ctrl in iter(self):
+            try:
+                if ctrl.run_io_seq(iocfg) is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def run_ios_random(self, iocfg):
+        """ Select a controller from the list of controllers
+            randomly and run IOs. Exhaust entire list.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - None.
+        """
+        ctrl_list = range(0, len(self.ctrl_list))
+
+        ret = True
+        for i in range(0, len(self.ctrl_list)):
+            random.shuffle(ctrl_list)
+            ctrl_id = ctrl_list.pop()
+            ctrl = self.ctrl_list[ctrl_id]
+            if ctrl.run_io_random(iocfg) is False:
+                ret = False
+                break
+        return ret
+
+    def ctrl_rescan(self):
+        """ Run controller_rescan on all host controllers sequentially.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            try:
+                if ctrl.ctrl_rescan() is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def ctrl_reset(self):
+        """ Run controller_reset on all host controllers sequentially.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            try:
+                if ctrl.ctrl_reset() is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def smart_log(self):
+        """ Execute smart log.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            if ctrl.smart_log() is False:
+                ret = False
+                break
+        return ret
+
+    def id_ctrl(self):
+        """ Execute id-ctrl on all the controllers(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            if ctrl.id_ctrl() is False:
+                ret = False
+                break
+        return ret
+
+    def id_ns(self):
+        """ Execute id-ns on controller(s) and all its namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            if ctrl.id_ns() is False:
+                ret = False
+                break
+        return ret
+
+    def ns_descs(self):
+        """ Execute ns-descs on all namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            if ctrl.ns_descs() is False:
+                ret = False
+                break
+        return ret
+
+    def get_ns_id(self):
+        """ Execute get-ns-id on all namespace(s).
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            if ctrl.get_ns_id() is False:
+                ret = False
+                break
+        return ret
+
+    def mkfs_seq(self, fs_type):
+        """ Run mkfs, mount fs and run IOs.
+            - Args :
+                - fs_type : file system type.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            try:
+                if ctrl.run_mkfs_seq(fs_type) is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def run_fs_ios(self, iocfg):
+        """ Run IOs on mounted file system.
+            - Args :
+                - iocfg : io configuration.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for ctrl in iter(self):
+            try:
+                if ctrl.run_fs_ios(iocfg) is False:
+                    ret = False
+                    break
+            except StopIteration:
+                break
+        return ret
+
+    def config(self, config_file="config/loop.json"):
+        """ Configure Host based on the target transport.
+            - Args :
+                - config_file : JSON config.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = False
+        if self.target_type == "loop":
+            self.logger.info("Configuring loop host")
+            ret = self.config_loop(config_file)
+            self.logger.info("Host configure successfully")
+        else:
+            self.logger.error("only loop target type is supported.")
+        return ret
+
+    def delete(self):
+        """ Delete all the Host Controllers.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        ret = True
+        for subsys in iter(self):
+            if subsys.delete() is False:
+                ret = False
+        return ret
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:41 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:41 -0700
Subject: [PATCH V2 24/46] nvmftests-tests: add a test for ns-descs
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-25-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test for namespace identification
descriptors command.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_ns_descs.py          | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ns_descs.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_descs.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_descs.py
new file mode 100644
index 0000000..3f3d04f
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_descs.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test ns-descs on each controller :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute ns-descs.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFNsDescs(NVMFTest):
+
+    """ Represents Identify Controller testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFNsDescs, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFNsDescs, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_ns_descs(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.ns_descs()
+        assert_equal(ret, True, "ERROR : id controller failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:38 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:38 -0700
Subject: [PATCH V2 21/46] nvmftests-tests: add support for test logger
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-22-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds support to store hierarchical logs for each execution.
Each test will use its unique class name under which this will
store all the logs.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/nvmf_test_logger.py  | 50 ++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/nvmf_test_logger.py

diff --git a/tools/testing/selftests/nvmftests/tests/nvmf_test_logger.py b/tools/testing/selftests/nvmftests/tests/nvmf_test_logger.py
new file mode 100644
index 0000000..16230f8
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/nvmf_test_logger.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""Logger for NVMe Test Framwwork.
+"""
+import sys
+
+
+class NVMFLogger(object):
+    """ Represents Logger for NVMe Testframework.  """
+    def __init__(self, log_file_path):
+        """ Logger setup.
+            - Args :
+                - log_file_path : path to store the log.
+        """
+        self.terminal = sys.stdout
+        self.log = open(log_file_path, "w")
+
+    def write(self, log_message):
+        """ Logger write.
+            - Args :
+                - log_message : string to write in the log file.
+            - Returns:
+                - None.
+        """
+        self.terminal.write(log_message)
+        self.log.write(log_message)
+
+    def flush(self):
+        """ This flush method is needed for python 3 compatibility.
+            this handles the flush command by doing nothing.
+            you might want to specify some extra behavior here.
+        """
+        pass
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:39 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:39 -0700
Subject: [PATCH V2 22/46] nvmftests-tests: add a parent class for tests
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-23-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a parent class for tests. Each test should inherit this
class and use common code present in the base class if possible.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../testing/selftests/nvmftests/tests/nvmf_test.py | 204 +++++++++++++++++++++
 1 file changed, 204 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/nvmf_test.py

diff --git a/tools/testing/selftests/nvmftests/tests/nvmf_test.py b/tools/testing/selftests/nvmftests/tests/nvmf_test.py
new file mode 100644
index 0000000..e1c155e
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/nvmf_test.py
@@ -0,0 +1,204 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+""" Base Class for all NVMF testcases.
+"""
+
+import os
+import sys
+import json
+from nose.tools import assert_equal
+
+from utils.const import Const
+from utils.diskio import DD
+from utils.diskio import FIO
+from nvmf.target import NVMFTarget
+from nvmf.host import NVMFHost
+from nvmf.target.target_config_generator import TargetConfig
+from nvmf_test_logger import NVMFLogger
+
+
+def __dd_worker__(iocfg):
+    """ dd worker thread function.
+        - Args :
+            - iocfg : io configuration.
+        - Returns :
+            - Return value of dd command.
+    """
+    return DD.run_io(iocfg)
+
+
+def __fio_worker__(iocfg):
+    """ fio worker thread function.
+        - Args :
+            - iocfg : io configuration.
+        - Returns :
+            - Return value of fio command.
+    """
+    return FIO.run_io(iocfg)
+
+
+class NVMFTest(object):
+
+    """ Common attributes for all testcases """
+
+    def __init__(self):
+        # TODO : make this dictionary
+        self.config_file = 'config/nvmftests.json'
+        self.data_size = 0
+        self.block_size = 0
+        self.nr_dev = 0
+        self.mount_path = Const.XXX
+        self.test_log_dir = Const.XXX
+        self.nr_dev = 0
+        self.nr_target_subsys = 0
+        self.nr_ns_per_subsys = 0
+        self.target_config_file = Const.XXX
+        self.target_type = Const.XXX
+        self.loopdev = None
+        self.host_subsys = None
+        self.target_subsys = None
+        self.log_dir = './logs/' + self.__class__.__name__ + '/'
+        self.err_str = "ERROR : " + self.__class__.__name__ + " : "
+        self.fio_read = {}
+        self.fio_fs_write = {}
+        self.dd_read = {}
+        self.dd_write = {}
+        self.blk_dev_pool = []
+
+        self.load_config()
+
+    def build_target_config(self, dev_list):
+        """ Generates target config file in JSON format from test config file.
+            - Args :
+                - None.
+            - Returns :
+                - None.
+        """
+        with open(self.config_file) as cfg_file:
+            cfg = json.load(cfg_file)
+            # target subsystem and namespsce config
+            self.nr_dev = int(cfg['nr_dev'])
+            self.nr_target_subsys = int(cfg['nr_target_subsys'])
+            self.nr_ns_per_subsys = int(cfg['nr_ns_per_subsys'])
+            self.target_config_file = cfg['target_config_file']
+            target_cfg = TargetConfig(self.target_config_file,
+                                      self.nr_target_subsys,
+                                      self.nr_ns_per_subsys,
+                                      dev_list)
+            target_cfg.build_target_subsys()
+
+    def load_config(self):
+        """ Load basic test configuration.
+            - Args :
+                - None.
+            - Returns :
+                - True on success, False on failure.
+        """
+        with open(self.config_file) as nvmftest_config:
+            cfg = json.load(nvmftest_config)
+            self.mount_path = cfg['mount_path']
+            self.data_size = self.human_to_bytes(cfg['data_size'])
+            self.block_size = self.human_to_bytes(cfg['block_size'])
+            self.nr_dev = int(cfg['nr_dev'])
+            # fio device read
+            self.fio_read = cfg['fio_read']
+            self.fio_read['filename'] = Const.XXX
+            self.fio_read['THREAD'] = __fio_worker__
+            self.fio_read['RC'] = 0
+            # fio file system write
+            self.fio_fs_write = cfg['fio_fs_write']
+            self.fio_fs_write['directory'] = Const.XXX
+            self.fio_fs_write['THREAD'] = __fio_worker__
+            self.fio_fs_write['RC'] = 0
+            # dd write
+            self.dd_read = cfg['dd_read']
+            self.dd_read['THREAD'] = __dd_worker__
+            self.dd_read['IF'] = None
+            self.dd_read['COUNT'] = str(self.data_size / self.block_size)
+            self.dd_read['RC'] = 0
+            # dd read
+            self.dd_write = cfg['dd_write']
+            self.dd_write['THREAD'] = __dd_worker__
+            self.dd_write['OF'] = None
+            self.dd_write['COUNT'] = str(self.data_size / self.block_size)
+            self.dd_write['RC'] = 0
+
+            self.target_type = cfg['target_type']
+            # block_dev_pool
+            self.blk_dev_pool = cfg['block_dev_pool']
+            return True
+        return False
+
+    def human_to_bytes(self, num_str):
+        """ Converts human readble format to bytes.
+            Args :
+              - num_str : human readble string.
+            Returns :
+              - On success decimal equivalant of num_str, 0 on failure.
+        """
+        decimal_bytes = 0
+        num_suffix = str(num_str[-2:]).upper()
+        if num_suffix == Const.KB:
+            decimal_bytes = int(num_str.split("K")[0]) * Const.ONE_KB
+        elif num_suffix == Const.MB:
+            decimal_bytes = int(num_str.split("M")[0]) * Const.ONE_MB
+        elif num_suffix == Const.GB:
+            decimal_bytes = int(num_str.split("G")[0]) * Const.ONE_GB
+        else:
+            print(self.err_str + "invalid suffix " + num_str)
+
+        return decimal_bytes
+
+    def setup_log_dir(self, test_name):
+        """ Set up the log directory for a testcase.
+            Args :
+              - test_name : name of the testcase.
+            Returns :
+              - None.
+        """
+        self.test_log_dir = self.log_dir + "/" + test_name
+        if not os.path.exists(self.test_log_dir):
+            os.makedirs(self.test_log_dir)
+        sys.stdout = NVMFLogger(self.test_log_dir + "/" + "stdout.log")
+        sys.stderr = NVMFLogger(self.test_log_dir + "/" + "stderr.log")
+
+    def common_setup(self):
+        """ Common test case setup function.
+            Args :
+              - None.
+            Returns :
+              - None.
+        """
+        self.target_subsys = NVMFTarget(self.target_type)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : target config failed")
+        self.host_subsys = NVMFHost(self.target_type)
+        ret = self.host_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : host config failed")
+
+    def common_tear_down(self):
+        """ Common test case tear down function.
+            Args :
+              - None.
+            Returns :
+              - None.
+        """
+        self.host_subsys.delete()
+        self.target_subsys.delete()
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:52 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:52 -0700
Subject: [PATCH V2 35/46] nvmftests-tests: add a test to run IOs parallely
 with dd
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-36-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to run IOs on all the controller
and its namespaces parallely from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_parallel_io.py       | 65 ++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_parallel_io.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_parallel_io.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_parallel_io.py
new file mode 100644
index 0000000..1b28e44
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_parallel_io.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test parallel IOs on all controllers :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Run parallel IOs on all available controller(s) and its namespace(s).
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFParallelIO(NVMFTest):
+
+    """ Represents Parallel Subsystem IO testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFParallelIO, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFParallelIO, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_parallel_io(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.run_ios_parallel(self.dd_read)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
+        ret = self.host_subsys.run_ios_parallel(self.dd_write)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:55 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:55 -0700
Subject: [PATCH V2 38/46] nvmftests-tests: add a test to run IOs with dd
 randomly
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-39-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to run IOs on all the controller
and its namespaces by randomly selecting the controller
from the host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_random_io.py         | 65 ++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_random_io.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_random_io.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_random_io.py
new file mode 100644
index 0000000..33bb4cb
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_random_io.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF randomly select a controller and run IOs :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Randomly select the controller and run IOs on all namespaces(s).
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFRandomIO(NVMFTest):
+
+    """ Represents Random Susbsytem IO testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFRandomIO, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFRandomIO, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_random_io(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.run_ios_random(self.dd_read)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
+        ret = self.host_subsys.run_ios_random(self.dd_write)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:54 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:54 -0700
Subject: [PATCH V2 37/46] nvmftests-tests: add a test to enable/disable target
 ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-38-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to enable/disable target namespace.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_ns_enable_disable.py | 83 ++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ns_enable_disable.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_enable_disable.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_enable_disable.py
new file mode 100644
index 0000000..94b521b
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_ns_enable_disable.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test enable disable target namespace :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Exercise enable disable target namespace options.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFNSEnableDisable(NVMFTest):
+
+    """ Represents enable disable target namespace testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFNSEnableDisable, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFNSEnableDisable, self).common_tear_down()
+        self.loopdev.delete()
+
+    def target_ns_enable_disable(self, target_ns):
+        target_ns_path_str = " Target NS Path " + target_ns.ns_path
+        print(" Target NS ID " + str(target_ns.ns_id))
+
+        ret = target_ns.disable()
+        assert_equal(ret, True, "ERROR : target ns enable failed ...")
+        print(target_ns_path_str + " disabled successfully")
+
+        ret = target_ns.enable()
+        assert_equal(ret, True, "ERROR : target ns disable failed ...")
+        print(target_ns_path_str + " enabled successfully")
+
+    def test_ns_enable_disable(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        for target_subsys in iter(self.target_subsys):
+            try:
+                print("Target Subsystem NQN " + target_subsys.nqn)
+                for target_ns in iter(target_subsys):
+                    try:
+                        self.target_ns_enable_disable(target_ns)
+                    except StopIteration:
+                        break
+            except StopIteration:
+                break
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:56 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:56 -0700
Subject: [PATCH V2 39/46] nvmftests-tests: add a test to scan host ctrls and ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-40-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to scan all the host ctrls and ns.
This can be used as a template in future tests where scanning
of the host subsystem is needed. e.g. adding a new ns after
connecting to the target and verifying that newly added ns
on target is appearing on the host in a timely manner.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_scan_host.py         | 76 ++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_scan_host.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_host.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_host.py
new file mode 100644
index 0000000..6ffd309
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_host.py
@@ -0,0 +1,76 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF scan host :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Scan host subsystem.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFScanHost(NVMFTest):
+
+    """ Represents host controller scan testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFScanHost, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFScanHost, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_scan_host(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        success = True
+        for host_subsys in iter(self.host_subsys):
+            try:
+                print("Host Controller " + host_subsys.ctrl_dev)
+                for host_ns in iter(host_subsys):
+                    try:
+                        print(" Host NS " + host_ns.ns_dev)
+                    except StopIteration:
+                        success = False
+                        break
+            except StopIteration:
+                success = False
+                break
+
+        assert_equal(success, True, "ERROR : failed to scan host")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:57 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:57 -0700
Subject: [PATCH V2 40/46] nvmftests-tests: add a test to scan target subsys/ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-41-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to scan all the target subsys and ns.
This can be used as a template where scanning of target subsystems
is needed while the test is in progress.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_scan_target.py       | 76 ++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_scan_target.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_target.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_target.py
new file mode 100644
index 0000000..b65e737
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_scan_target.py
@@ -0,0 +1,76 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF scan Target :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Scan target subsystem.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFScanTarget(NVMFTest):
+
+    """ Represents host controller scan testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFScanTarget, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFScanTarget, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_scan_target(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        success = True
+        for target_subsys in iter(self.target_subsys):
+            try:
+                print("Target Controller " + target_subsys.nqn)
+                for target_ns in iter(target_subsys):
+                    try:
+                        print(" Target NS " + target_ns.ns_path)
+                    except StopIteration:
+                        success = False
+                        break
+            except StopIteration:
+                success = False
+                break
+
+        assert_equal(success, True, "ERROR : failed to scan host")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:58 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:58 -0700
Subject: [PATCH V2 41/46] nvmftests-tests: add a target template
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-42-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a target test template which can be easily copied
to write a new testcase.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_target_template.py   | 64 ++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_target_template.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_target_template.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_target_template.py
new file mode 100644
index 0000000..08f0978
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_target_template.py
@@ -0,0 +1,64 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF target template :-
+
+    1. From the config file create Target.
+    2. Write testcase code here.
+    3. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+from nvmf.target import NVMFTarget
+
+
+class TestNVMFTargetTemplate(NVMFTest):
+
+    """ Represents target template testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.target_subsys = None
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        self.target_subsys = NVMFTarget(self.target_type)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : target config failed")
+
+    def tearDown(self):
+        """ Post section of testcase """
+        self.target_subsys.delete()
+        self.loopdev.delete()
+
+    def test_target(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        assert_equal(0, 0, "")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:31:03 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:31:03 -0700
Subject: [PATCH V2 46/46] nvmftests: add Makefile
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-47-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 tools/testing/selftests/nvmftests/Makefile | 63 ++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/Makefile

diff --git a/tools/testing/selftests/nvmftests/Makefile b/tools/testing/selftests/nvmftests/Makefile
new file mode 100644
index 0000000..ecb523a
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/Makefile
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+
+help: all
+
+all:
+	@echo "Usage:"
+	@echo
+	@echo "  make run         - Run all testcases."
+	@echo "  make doc         - Generate Documentation."
+	@echo "  make cleanall    - removes *pyc, documentation."
+	@echo "  make static_check- runs pep8, flake8, and pylint on code."
+
+doc:
+	@make -C doc/
+
+run:
+	@ln -s tests/config config
+	nose2 --verbose
+	@rm -fr config
+
+static_check:
+	for i in `find . -name \*.py  | grep -v __init__ | grep -v state_machine`;\
+	do\
+			echo "Pylint :- " ;\
+			printf "%10s    " $${i};\
+			pylint $${i} 2>&1  | grep "^Your code" |  awk '{print $$7}';\
+			echo "--------------------------------------------";\
+			echo "pep8 :- ";\
+			pep8 $${i};\
+			echo "flake8 :- ";\
+			flake8 $${i};\
+	done
+
+cleanall: clean
+	@rm -fr tests/logs tests/*fio.log loop.json logs
+	@find . -name \*_fio.log | xargs rm -fr
+	@make -C doc/ clean
+
+clean:
+	@rm -fr config
+	@find . -name \*pyc | xargs rm -fr
+	@find . -name __pycache__ | xargs rm -fr
+	@find . -name \*ropeproject | xargs rm -fr
+
+.PHONY: doc clean cleanall
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:31:02 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:31:02 -0700
Subject: [PATCH V2 45/46] nvmftests: add class documentation and sequence
 diagram
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-46-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 tools/testing/selftests/nvmftests/doc/Makefile     |  7 ++
 .../selftests/nvmftests/doc/sequence-diag/Makefile |  7 ++
 .../nvmftests/doc/sequence-diag/nvmftests.plantuml | 98 ++++++++++++++++++++++
 3 files changed, 112 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/doc/Makefile
 create mode 100644 tools/testing/selftests/nvmftests/doc/sequence-diag/Makefile
 create mode 100644 tools/testing/selftests/nvmftests/doc/sequence-diag/nvmftests.plantuml

diff --git a/tools/testing/selftests/nvmftests/doc/Makefile b/tools/testing/selftests/nvmftests/doc/Makefile
new file mode 100644
index 0000000..868b34a
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/doc/Makefile
@@ -0,0 +1,7 @@
+doc:
+	@epydoc -v --output=Documentation ../nvmf/*py ../tests/*py ../utils/*py
+	@make -C sequence-diag/
+
+clean:
+	@make -C sequence-diag/ clean
+	@rm -fr Documentation
diff --git a/tools/testing/selftests/nvmftests/doc/sequence-diag/Makefile b/tools/testing/selftests/nvmftests/doc/sequence-diag/Makefile
new file mode 100644
index 0000000..4a25c61
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/doc/sequence-diag/Makefile
@@ -0,0 +1,7 @@
+SRC=nvmftests
+
+all:
+	@plantuml ${SRC}.plantuml
+
+clean:
+	@rm -fr ${SRC}.png
diff --git a/tools/testing/selftests/nvmftests/doc/sequence-diag/nvmftests.plantuml b/tools/testing/selftests/nvmftests/doc/sequence-diag/nvmftests.plantuml
new file mode 100644
index 0000000..f3df765
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/doc/sequence-diag/nvmftests.plantuml
@@ -0,0 +1,98 @@
+ at startuml
+autonumber
+
+title nvmftest Sequence
+
+box "testcase.py" #LightBlue
+participant "Testcase" as tc
+end box
+
+box "target.py " #LightYellow
+participant "Target subsystems" as t
+end box
+
+box "target_subsystem.py" #LightBlue
+participant "Target namespace" as ts
+
+end box
+
+box "target_ns.py" #Gray
+participant "target namespace processing" as tns
+end box
+
+activate tc
+tc->t : initialize target object
+
+activate t
+t->ts : read config file and create target subsystem config_loop_target()
+ts->ts : initialize the target subsystem init()
+activate ts
+
+activate tns
+tns->tns : create and initialize namespace(s) init()
+tns->tns : initialize with device nguid
+tns->tns : initialize with device path
+tns->tns : enable ns enable_ns()
+
+tns->ts :
+ts->t :
+t->t : initialize port and assign subsystems to the port init()
+t->tc
+
+box "host.py " #LightYellow
+participant "host subsystems" as h
+end box
+
+box "host_subsystem.py" #LightBlue
+participant "host Namespace" as hs
+end box
+
+box "host_ns.py" #Gray
+participant "host namespace processing" as hns
+end box
+
+activate tc
+tc->h : initialize and configure host config()
+
+activate h
+h->hs : read config file and create host subsystem config_loop()
+hs->hs : initialize the host subsystem, connect to target init_ctrl()
+
+activate hs
+hs->hns : create and initialize namespace init_ctrl_ns()
+hns->hns : validate sysfs entries
+
+activate hns
+hns->hns : validate host namespace as a block device init()
+hns->hns : validate sysfs entries validate_sysfs_ns()
+
+hns->hs :
+hs->h :
+h->tc
+
+tc->tc : execute test case operations
+
+tc->h : delete host delete()
+h->hs : delete host system delete()
+hs->hns : delete host namespace delete()
+
+hns->hs :
+deactivate hns
+hs->h :
+deactivate hs
+h->tc
+deactivate h
+
+tc->t : delete target delete()
+t->t : delete port delete()
+t->ts : delete target subsystem delete()
+ts->tns : delete target namespaces delete()
+tns->ts
+deactivate tns
+deactivate ts
+ts->t
+deactivate t
+t->tc
+deactivate tc
+
+ at enduml
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:51 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:51 -0700
Subject: [PATCH V2 34/46] nvmftests-tests: add a test to run IOs with dd
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-35-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to run IOs on all the controllers
and its namespaces. At a time it selects only one namespace
to run IOs.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/test_nvmf_io.py      | 65 ++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_io.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_io.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_io.py
new file mode 100644
index 0000000..b1ccaa8
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_io.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF sequntially select a controller and run IOs:-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Run IOs sequentially iterating over controller(s) and its namespaces(s).
+    3. Delete Host.
+    4. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFIO(NVMFTest):
+
+    """ Represents Sequential Subsystem IO testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFIO, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFIO, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_io(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.run_ios_seq(self.dd_read)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
+        ret = self.host_subsys.run_ios_seq(self.dd_write)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:50 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:50 -0700
Subject: [PATCH V2 33/46] nvmftests-tests: add a test for smart-log
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-34-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute smart-log command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_smart_log.py         | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_smart_log.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_smart_log.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_smart_log.py
new file mode 100644
index 0000000..952ca04
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_smart_log.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test smart log on all controllers :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute smart-log on all controllers and its namespaces.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFSmartLog(NVMFTest):
+
+    """ Represents Smart Log testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFSmartLog, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFSmartLog, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_smart_log(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.smart_log()
+        assert_equal(ret, True, "ERROR : running smart log failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:46 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:46 -0700
Subject: [PATCH V2 29/46] nvmftests-tests: add a test for controller reset
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-30-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute controller reset command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_ctrl_reset.py        | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_reset.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_reset.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_reset.py
new file mode 100644
index 0000000..acad21c
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_reset.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF host controller reset :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Call controller reset on each host controller.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFCtrlReset(NVMFTest):
+
+    """ Represents host controller reset testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFCtrlReset, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFCtrlReset, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_ctrl_reset(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.ctrl_reset()
+        assert_equal(ret, True, "ERROR : ctrl reset failed")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:44 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:44 -0700
Subject: [PATCH V2 27/46] nvmftests-tests: add a test to create and delete
 target
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-28-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to create and delete target with loopback
device.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_create_target.py     | 62 ++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_target.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_create_target.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_target.py
new file mode 100644
index 0000000..4f2497c
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_target.py
@@ -0,0 +1,62 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF Create/Delete Target :-
+
+    1. From the config file create Target.
+    2. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+from nvmf.target import NVMFTarget
+
+
+class TestNVMFCreateTarget(NVMFTest):
+
+    """ Represents Create Target testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.target_subsys = None
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        self.target_subsys = NVMFTarget(self.target_type)
+
+    def tearDown(self):
+        """ Post section of testcase """
+        self.target_subsys.delete()
+        self.loopdev.delete()
+
+    def test_create_target(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : config target failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:43 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:43 -0700
Subject: [PATCH V2 26/46] nvmftests-tests: add a test for NVMe PCIe target ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-27-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to create and delete host with NVMe
PCIe block device on the target side.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_create_pci_target.py | 70 ++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_pci_target.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_create_pci_target.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_pci_target.py
new file mode 100644
index 0000000..f1a8a5f
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_pci_target.py
@@ -0,0 +1,70 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF Create/Delete Host with PCIe device on target namespace :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Delete Host.
+    4. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.nvme_pci import NVMePCIeBlk
+from nvmf_test import NVMFTest
+from nvmf.target import NVMFTarget
+from nvmf.host import NVMFHost
+
+
+class TestNVMFCreatePCIeTarget(NVMFTest):
+
+    """ Represents Host Creation testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.host_subsys = None
+        self.target_subsys = None
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.nvme_pci = NVMePCIeBlk()
+        self.nvme_pci.init()
+        self.build_target_config(self.nvme_pci.dev_list)
+        self.target_subsys = NVMFTarget(self.target_type)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : target config failed")
+        self.host_subsys = NVMFHost(self.target_type)
+
+    def tearDown(self):
+        """ Post section of testcase """
+        self.host_subsys.delete()
+        self.target_subsys.delete()
+        self.nvme_pci.delete()
+
+    def test_create_pci_target(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : host config failed")
+        raw_input("Press enter to continue ...")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:37 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:37 -0700
Subject: [PATCH V2 20/46] nvmftests-tests: add user config file
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-21-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a centralized config file which can be used to
specify different parameters for the execution of the test.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/config/nvmftests.json          | 83 ++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/config/nvmftests.json

diff --git a/tools/testing/selftests/nvmftests/tests/config/nvmftests.json b/tools/testing/selftests/nvmftests/tests/config/nvmftests.json
new file mode 100644
index 0000000..7dccdad
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/config/nvmftests.json
@@ -0,0 +1,83 @@
+{
+	"mount_path" : "/mnt/",
+	"data_size"  : "128MB",
+	"block_size" : "4KB",
+	"nr_dev" : "5",
+	"nr_target_subsys" : "1",
+	"nr_ns_per_subsys" : "1",
+	"target_config_file" : "loop.json",
+	"target_type" : "loop",
+	"fio_read": {
+		"IO_TYPE": "fio",
+		"group_reporting": "1",
+		"rw": "randread",
+		"bs": "4k",
+		"numjobs": "4",
+		"iodepth": "8",
+		"runtime": "30",
+		"loop": "1",
+		"ioengine": "libaio",
+		"direct": "1",
+		"invalidate": "1",
+		"randrepeat": "1",
+		"size": "100M",
+		"filename": "XXX",
+		"name": "test1",
+		"THREAD": "XXX",
+		"RC": "0"
+	},
+	"fio_fs_write": {
+		"IO_TYPE": "fio",
+		"group_reporting": "1",
+		"rw": "randwrite",
+		"bs": "4k",
+		"numjobs": "4",
+		"iodepth": "8",
+		"runtime": "30",
+		"loop": "1",
+		"ioengine": "libaio",
+		"direct": "1",
+		"invalidate": "1",
+		"randrepeat": "1",
+		"size": "10M",
+		"directory": "XXX",
+		"name": "test1",
+		"THREAD": "XXX",
+		"RC": "0"
+	},
+	"dd_read": {
+		"IO_TYPE": "dd",
+		"IODIR": "read",
+		"THREAD": "XXX",
+		"IF": "XXX",
+		"OF": "/dev/null",
+		"BS": "4K",
+		"COUNT": "XXX",
+		"RC": "0"
+	},
+	"dd_write": {
+		"IO_TYPE": "dd",
+		"IODIR": "write",
+		"THREAD": "XXX",
+		"IF": "/dev/zero",
+		"OF": "XXX",
+		"BS": "4K",
+		"COUNT": "XXX",
+		"RC": "0"
+	},
+	"block_dev_pool": [
+		"/dev/nvme0n1"
+	],
+	"log": {
+		"host" : "DEBUG",
+		"host_subsystem" : "DEBUG",
+		"host_ns" : "DEBUG",
+		"target" : "DEBUG",
+		"target_subsystem" : "DEBUG",
+		"target_ns" : "DEBUG",
+		"port" : "DEBUG",
+		"nvme_pci": "DEBUG",
+		"loopback": "DEBUG",
+		"gen_blk": "DEBUG"
+	}
+}
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:47 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:47 -0700
Subject: [PATCH V2 30/46] nvmftests-tests: add a test for get-ns-id
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-31-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute get-ns-id command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_get_ns_id.py         | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_get_ns_id.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_get_ns_id.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_get_ns_id.py
new file mode 100644
index 0000000..9e49a08
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_get_ns_id.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test get-ns-id on each controller :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute get-ns-id on all controllers.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFGetNsId(NVMFTest):
+
+    """ Represents Identify Controller testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFGetNsId, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFGetNsId, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_get_nsid(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.get_ns_id()
+        assert_equal(ret, True, "ERROR : id controller failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:40 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:40 -0700
Subject: [PATCH V2 23/46] nvmftests-tests: add a test for generic block device
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-24-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new testcase to create target and host, on the
target side it uses generic block device.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_create_gen_blk.py    | 71 ++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_gen_blk.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_create_gen_blk.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_gen_blk.py
new file mode 100644
index 0000000..7ee4eab
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_gen_blk.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF Create/Delete Host using generic block device :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Delete Host.
+    4. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.generic_blk_dev import GenBlk
+from nvmf_test import NVMFTest
+from nvmf.target import NVMFTarget
+from nvmf.host import NVMFHost
+
+
+class TestNVMFCreateGenBlk(NVMFTest):
+
+    """ Represents Host Creation testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.host_subsys = None
+        self.target_subsys = None
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.gendev = GenBlk(len(self.blk_dev_pool))
+        self.gendev.init(self.blk_dev_pool)
+
+        self.build_target_config(self.gendev.dev_list)
+
+        self.target_subsys = NVMFTarget(self.target_type)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : target config failed")
+
+        self.host_subsys = NVMFHost(self.target_type)
+
+    def tearDown(self):
+        """ Post section of testcase """
+        self.host_subsys.delete()
+        self.target_subsys.delete()
+
+    def test_create_gen_blk(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : host config failed")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:31:00 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:31:00 -0700
Subject: [PATCH V2 43/46] nvmftests-tests: add a test to run traffic and
 disable ns
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-44-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to run the IOs from the host
and disable the ns on the target.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_run_host_traffic.py  | 111 +++++++++++++++++++++
 1 file changed, 111 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_run_host_traffic.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_run_host_traffic.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_run_host_traffic.py
new file mode 100644
index 0000000..bf057d0
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_run_host_traffic.py
@@ -0,0 +1,111 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF run host traffic and disable target namespace(s) :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Start host traffic parallely on all host ns, disable target ns.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import time
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.diskio import DD
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+def __run_traffic__(iocfg):
+    """ dd worker thread function to run IOs until it fails.
+        - Args :
+              - iocfg : io configuration.
+        - Returns :
+              - True when dd command fails.
+    """
+    print("Run traffic :- ")
+    while True:
+        ret = DD.run_io(iocfg)
+        # For this testcase we need dd to fail after disabling the ns
+        if ret is False:
+            return True
+
+
+class TestNVMFHostTraffic(NVMFTest):
+
+    """ Represents disable target namespace(s) while host traffic is running
+        testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        self.dd_read_traffic = {"IO_TYPE": "dd",
+                                "IODIR": "read",
+                                "THREAD": __run_traffic__,
+                                "IF": None,
+                                "OF": "/dev/null",
+                                "BS": "4K",
+                                "COUNT": str(self.data_size / self.block_size),
+                                "RC": 0}
+        super(TestNVMFHostTraffic, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFHostTraffic, self).common_tear_down()
+        self.loopdev.delete()
+
+    def disable_target_ns(self):
+        ret = True
+        for target_subsys in iter(self.target_subsys):
+            try:
+                print("Target Subsystem NQN " + target_subsys.nqn)
+                for target_ns in iter(target_subsys):
+                    try:
+                        ns_path = target_ns.ns_path
+                        print(" Target NS ID " + str(target_ns.ns_id))
+                        print(" Disabling Target NS Path " + ns_path)
+                        if target_ns.disable() is False:
+                            print("ERROR : failed to disable ns " + ns_path)
+                            ret = False
+                    except StopIteration:
+                        break
+            except StopIteration:
+                break
+        return ret
+
+    def test_host_traffic(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        self.host_subsys.run_traffic_parallel(self.dd_read_traffic)
+        time.sleep(5)
+        ret = self.disable_target_ns()
+        assert_equal(ret, True, "ERROR : target ns disable failed")
+        self.host_subsys.wait_traffic_parallel()
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:31:01 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:31:01 -0700
Subject: [PATCH V2 44/46] nvmftests-tests: add a test to measure perf with fio
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-45-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test which runs fio to measure performance
with simple workload.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/test_nvmf_perf.py    | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_perf.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_perf.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_perf.py
new file mode 100644
index 0000000..853b5558
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_perf.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test parallel IOs on all controllers :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Run parallel IOs on all available controller(s) and its namespace(s).
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFParallelPerf(NVMFTest):
+
+    """ Represents Parallel Subsystem IO testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFParallelPerf, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFParallelPerf, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_parallel_perf(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.run_perf_parallel(self.fio_read)
+        assert_equal(ret, True, "ERROR : running IOs failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:59 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:59 -0700
Subject: [PATCH V2 42/46] nvmftests-tests: add a host template
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-43-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a host test template which can be easily copied
to write a new testcase.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_host_template.py     | 62 ++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_host_template.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_host_template.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_host_template.py
new file mode 100644
index 0000000..bd3e50b
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_host_template.py
@@ -0,0 +1,62 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF host template :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Write testcase code in the testcase function.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFHostTemplate(NVMFTest):
+
+    """ Represents host template testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFHostTemplate, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFHostTemplate, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_host(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        assert_equal(0, 0, "")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:53 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:53 -0700
Subject: [PATCH V2 36/46] nvmftests-tests: add a test to run mkfs and fio
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-37-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to format host namespace(s) with mkfs,
mount the file system and run fio.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/test_nvmf_mkfs.py    | 65 ++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_mkfs.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_mkfs.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_mkfs.py
new file mode 100644
index 0000000..6f78e86
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_mkfs.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test mkfs on each subsystem :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute mkfs on all host namespaces.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFMKFS(NVMFTest):
+
+    """ Represents mkfs testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFMKFS, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFMKFS, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_mkfs(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.mkfs_seq("ext4")
+        assert_equal(ret, True, "ERROR : mkfs failed.")
+        ret = self.host_subsys.run_fs_ios(self.fio_fs_write)
+        assert_equal(ret, True, "ERROR : fio failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:49 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:49 -0700
Subject: [PATCH V2 32/46] nvmftests-tests: add a test for identify namespace
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-33-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute identify namespace command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/test_nvmf_id_ns.py   | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_id_ns.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ns.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ns.py
new file mode 100644
index 0000000..365bcad
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ns.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test id-ns on each namespace :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute id-ns.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFIdentifyNamespace(NVMFTest):
+
+    """ Represents Identify Namespace Test testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFIdentifyNamespace, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFIdentifyNamespace, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_identify_namespace(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.id_ns()
+        assert_equal(ret, True, "ERROR : id ns failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:48 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:48 -0700
Subject: [PATCH V2 31/46] nvmftests-tests: add a test for identify controller
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-32-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute identify controller command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../selftests/nvmftests/tests/test_nvmf_id_ctrl.py | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_id_ctrl.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ctrl.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ctrl.py
new file mode 100644
index 0000000..37acd76
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_id_ctrl.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF test id-ctrl on each controller :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Execute id-ctrl on all controllers.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFIdentifyController(NVMFTest):
+
+    """ Represents Identify Controller testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFIdentifyController, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFIdentifyController, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_identify_controller(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.id_ctrl()
+        assert_equal(ret, True, "ERROR : id controller failed.")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:45 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:45 -0700
Subject: [PATCH V2 28/46] nvmftests-tests: add a test for controller rescan
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-29-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to execute controller rescan command
from host.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_ctrl_rescan.py       | 63 ++++++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_rescan.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_rescan.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_rescan.py
new file mode 100644
index 0000000..f75f443
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_ctrl_rescan.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF host controller rescan :-
+
+    1. From config file create Target.
+    2. From config file create host and connect to target.
+    3. Call controller rescan on each host controller.
+    4. Delete Host.
+    5. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+
+
+class TestNVMFCtrlRescan(NVMFTest):
+
+    """ Represents host controller rescan testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        super(TestNVMFCtrlRescan, self).common_setup()
+
+    def tearDown(self):
+        """ Post section of testcase """
+        super(TestNVMFCtrlRescan, self).common_tear_down()
+        self.loopdev.delete()
+
+    def test_ctrl_rescan(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.ctrl_rescan()
+        assert_equal(ret, True, "ERROR : ctrl rescan failed")
-- 
1.8.3.1



From ckulkarnilinux at gmail.com  Tue Oct 24 18:30:42 2017
From: ckulkarnilinux at gmail.com (Chaitanya Kulkarni)
Date: Tue, 24 Oct 2017 18:30:42 -0700
Subject: [PATCH V2 25/46] nvmftests-tests: add a test for create/delete host
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <1508895063-6280-26-git-send-email-ckulkarnilinux@gmail.com>

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This adds a new test to create and delete host with loopback
device.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 .../nvmftests/tests/test_nvmf_create_host.py       | 71 ++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/nvmftests/tests/test_nvmf_create_host.py

diff --git a/tools/testing/selftests/nvmftests/tests/test_nvmf_create_host.py b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_host.py
new file mode 100644
index 0000000..1f69535
--- /dev/null
+++ b/tools/testing/selftests/nvmftests/tests/test_nvmf_create_host.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2016-2017 Western Digital Corporation or its affiliates.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.
+#
+#   Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
+#
+"""
+NVMF Create/Delete Host :-
+
+    1. From the config file create Target.
+    2. From the config file create host and connect to target.
+    3. Delete Host.
+    4. Delete Target.
+"""
+
+
+import sys
+from nose.tools import assert_equal
+sys.path.append("../")
+from utils.misc.loopback import Loopback
+from nvmf_test import NVMFTest
+from nvmf.target import NVMFTarget
+from nvmf.host import NVMFHost
+
+
+class TestNVMFCreateHost(NVMFTest):
+
+    """ Represents Host Creation testcase """
+
+    def __init__(self):
+        NVMFTest.__init__(self)
+        self.host_subsys = None
+        self.target_subsys = None
+        self.setup_log_dir(self.__class__.__name__)
+
+    def setUp(self):
+        """ Pre section of testcase """
+        self.loopdev = Loopback(self.mount_path, self.data_size,
+                                self.block_size, self.nr_dev)
+        self.loopdev.init()
+        self.build_target_config(self.loopdev.dev_list)
+        self.target_subsys = NVMFTarget(self.target_type)
+        ret = self.target_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : target config failed")
+        self.host_subsys = NVMFHost(self.target_type)
+
+    def tearDown(self):
+        """ Post section of testcase """
+        self.host_subsys.delete()
+        self.target_subsys.delete()
+        self.loopdev.delete()
+
+    def test_create_host(self):
+        """ Testcase main """
+        print("Now Running " + self.__class__.__name__)
+        ret = self.host_subsys.config(self.target_config_file)
+        assert_equal(ret, True, "ERROR : host config failed")
+        raw_input("Press enter to continue ...")
-- 
1.8.3.1



From jthumshirn at suse.de  Wed Oct 25 00:30:32 2017
From: jthumshirn at suse.de (Johannes Thumshirn)
Date: Wed, 25 Oct 2017 09:30:32 +0200
Subject: [PATCH V2 00/46] nvmftests: NVMe Over Fabrics Unit Test Framework
In-Reply-To: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
 (Chaitanya Kulkarni's message of "Tue, 24 Oct 2017 18:30:17 -0700")
References: <1508895063-6280-1-git-send-email-ckulkarnilinux@gmail.com>
Message-ID: <mqda80fn0pz.fsf@linux-x5ow.site>

Hi Chaitanya,

thanks for doing this. I'll give it a spin in my NVMf test setup.

Byte,
        Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


From jsmart2021 at gmail.com  Wed Oct 25 07:41:30 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 07:41:30 -0700
Subject: [REPOST PATCH v2] nvmet: Fix fatal_err_work deadlock
Message-ID: <20171025144130.8833-1-jsmart2021@gmail.com>

Below is  a stack trace for an issue that was reported.

What's happening is that the nvmet layer had it's controller kato
timeout fire, which causes it to schedule its fatal error handler
via the fatal_err_work element. The error handler is invoked, which
calls the transport delete_ctrl() entry point, and as the transport
tears down the controller, nvmet_sq_destroy ends up doing the final
put on the ctlr causing it to enter its free routine. The ctlr free
routine does a cancel_work_sync() on fatal_err_work element, which
then does a flush_work and wait_for_completion. But, as the wait is
in the context of the work element being flushed, its in a catch-22
and the thread hangs.

[  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  326.909832] nvmet: ctrl 1 fatal error occurred!
[  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
x114 flg x2
[  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
seconds.
[  494.589638]       Not tainted 4.14.0-rc1.James+ #1
[  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  494.603718] kworker/0:2     D    0   243      2 0x80000000
[  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
[  494.616447] Call Trace:
[  494.619177]  __schedule+0x28d/0x890
[  494.623070]  schedule+0x36/0x80
[  494.626571]  schedule_timeout+0x1dd/0x300
[  494.631044]  ? dequeue_task_fair+0x592/0x840
[  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
[  494.640756]  wait_for_completion+0x121/0x180
[  494.645521]  ? wake_up_q+0x80/0x80
[  494.649315]  flush_work+0x11d/0x1a0
[  494.653206]  ? wake_up_worker+0x30/0x30
[  494.657484]  __cancel_work_timer+0x10b/0x190
[  494.662249]  cancel_work_sync+0x10/0x20
[  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
[  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
[  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
[  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
[  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
[  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
[  494.701589]  process_one_work+0x149/0x360
[  494.706064]  worker_thread+0x4d/0x3c0
[  494.710148]  kthread+0x109/0x140
[  494.713751]  ? rescuer_thread+0x380/0x380
[  494.718214]  ? kthread_park+0x60/0x60

Correct by creating a final free work element, which is scheduled
by the final ctrl put routine, so that when the flush_work (was
cancel_work_sync) is called, it cannot be in the same work_q
element context.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
v2:
  convert cancel_work_sync() to flush_work() for fatal_err_work
v3: cancelled. v2 is correct

 drivers/nvme/target/core.c  | 28 +++++++++++++++++++---------
 drivers/nvme/target/nvmet.h |  1 +
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index da088293eafc..ed7f30c45118 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -853,19 +853,14 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	return status;
 }
 
-static void nvmet_ctrl_free(struct kref *ref)
+static void nvmet_ctrl_free_work(struct work_struct *work)
 {
-	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_ctrl *ctrl =
+			container_of(work, struct nvmet_ctrl, free_work);
 	struct nvmet_subsys *subsys = ctrl->subsys;
 
-	nvmet_stop_keep_alive_timer(ctrl);
-
-	mutex_lock(&subsys->lock);
-	list_del(&ctrl->subsys_entry);
-	mutex_unlock(&subsys->lock);
-
 	flush_work(&ctrl->async_event_work);
-	cancel_work_sync(&ctrl->fatal_err_work);
+	flush_work(&ctrl->fatal_err_work);
 
 	ida_simple_remove(&cntlid_ida, ctrl->cntlid);
 	nvmet_subsys_put(subsys);
@@ -875,6 +870,21 @@ static void nvmet_ctrl_free(struct kref *ref)
 	kfree(ctrl);
 }
 
+static void nvmet_ctrl_free(struct kref *ref)
+{
+	struct nvmet_ctrl *ctrl = container_of(ref, struct nvmet_ctrl, ref);
+	struct nvmet_subsys *subsys = ctrl->subsys;
+
+	nvmet_stop_keep_alive_timer(ctrl);
+
+	mutex_lock(&subsys->lock);
+	list_del(&ctrl->subsys_entry);
+	mutex_unlock(&subsys->lock);
+
+	INIT_WORK(&ctrl->free_work, nvmet_ctrl_free_work);
+	schedule_work(&ctrl->free_work);
+}
+
 void nvmet_ctrl_put(struct nvmet_ctrl *ctrl)
 {
 	kref_put(&ctrl->ref, nvmet_ctrl_free);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index e342f02845c1..ed38b44a7007 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -129,6 +129,7 @@ struct nvmet_ctrl {
 	struct kref		ref;
 	struct delayed_work	ka_work;
 	struct work_struct	fatal_err_work;
+	struct work_struct	free_work;
 
 	struct nvmet_fabrics_ops *ops;
 
-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:12 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:12 -0700
Subject: [PATCH v4 0/5] nvme_fc: add dev_loss_tmo support
Message-ID: <20171025234317.13302-1-jsmart2021@gmail.com>

FC, on the SCSI side, has long had a device loss timeout which governed
how long it would hide connectivity loss to remote target ports.
The timeout value is maintained in the SCSI FC transport and admins
are used to going there to maintain it.

Eventually, the SCSI FC transport will be moved into something
independent from and above SCSI so that SCSI and NVME protocols can
be peers. In the meantime, to add the functionality now, and sync with
the SCSI FC transport, the LLDD will be used as the conduit. The
initial value for the timeout can be set by the LLDD when it creates
the remoteport via nvme_fc_register_remoteport(). Later, if the value
is updated via the SCSI transport, the LLDD can call a new nvme_fc
routine to update the remoteport's dev_loss_tmo value.

The nvme fabrics implementation already has a similar timer, the
ctrl_loss_tmo, which is distilled into a max_reconnect attempts and
a reconnect_delay between attempts, where the overall duration until
max is hit is the ctrl_loss_tmo.  This was primarily for transports
that didn't have the ability to track device connectivity and would
retry per the delay until finally giving up.

The implementation in this patch set maintains a FC dev_loss_tmo value
at the FC port level. When connectivity to a remoteport is lost, the
future time where dev_loss_tmo will expire is set, and all controllers
on the remoteport are suspended and their associations terminated.
The termination of the controllers will cause their ctrl_loss_tmo
functionality kick in. Reconnect attempts that occur while connectivity
is still lost are terminated and the next reconnect scheduled. If a
reconnect would be rescheduled for a time exceeding the dev_loss_tmo
for the remoteport, the next reconnect is scheduled at dev_loss_tmo.

If connectivity is re-established before ctrl_loss_tmo expires or the
dev_loss_tmo time expires, then the controller is immediately reconnected
and resumed.

If connectivity is not re-established before ctrl_loss_tmo expires or
the dev_loss_tmo time expires, then the controller is deleted.

The patches were cut on the nvme-4.15 branch
Patch 5, which adds the dev_loss_tmo timeout, is dependent on the
nvme_fc_signal_discovery_scan() routine added by this patch:
http://lists.infradead.org/pipermail/linux-nvme/2017-September/012781.html
The patch has been approved but not yet pulled into a tree.

v3:
 In v2, the implementation merged the dev_loss_tmo value into the
 ctlr_loss_tmo in the controller, so only a single timer on each controller
 was running.
 V3 changed to keep the dev_loss_tmo on the FC remoteport and to run it
 independently from the ctrl_loss_tmo timer, excepting for loss of
 connectivity to start both simultaneously.
v4:
 removed the dev_loss_tmo timer on the remote port object. Instead, add
 dev_loss_tmo as a time cap for ctrl_loss_tmo (but now not trashing the
 ctrl_loss_tmo values like early version patches). Thus dev_loss_tmo
 is enforced on a per-controller basis.


James Smart (5):
  nvme core: allow controller RESETTING to RECONNECTING transition
  nvme_fc: change ctlr state assignments during reset/reconnect
  nvme_fc: add a dev_loss_tmo field to the remoteport
  nvme_fc: check connectivity before initiating reconnects
  nvme_fc: add dev_loss_tmo timeout and remoteport resume support

 drivers/nvme/host/core.c       |   1 +
 drivers/nvme/host/fc.c         | 321 ++++++++++++++++++++++++++++++++++++-----
 include/linux/nvme-fc-driver.h |  11 +-
 3 files changed, 291 insertions(+), 42 deletions(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:13 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:13 -0700
Subject: [PATCH v4 1/5] nvme core: allow controller RESETTING to RECONNECTING
 transition
In-Reply-To: <20171025234317.13302-1-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
Message-ID: <20171025234317.13302-2-jsmart2021@gmail.com>

Allow controller state transition : RESETTING to RECONNECTING

Transport will typically transition from LIVE->RESETTING when
initially performing a reset or recovering from an error. Adding
this transition allows a transport to transition to RECONNECTING
when it checks/waits for connectivity then creates new transport
connections and reinits the controller.

-- james

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
Reviewed-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7fae42d595d5..04f0c7e1e5a2 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -207,6 +207,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 	case NVME_CTRL_RECONNECTING:
 		switch (old_state) {
 		case NVME_CTRL_LIVE:
+		case NVME_CTRL_RESETTING:
 			changed = true;
 			/* FALLTHRU */
 		default:
-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:16 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:16 -0700
Subject: [PATCH v4 4/5] nvme_fc: check connectivity before initiating
 reconnects
In-Reply-To: <20171025234317.13302-1-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
Message-ID: <20171025234317.13302-5-jsmart2021@gmail.com>

check remoteport connectivity before initiating reconnects

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
Reviewed-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/fc.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 0e10a24a8786..78ec1d47d1fc 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -810,7 +810,6 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
 		dma_unmap_sg(dev, sg, nents, dir);
 }
 
-
 /* *********************** FC-NVME LS Handling **************************** */
 
 static void nvme_fc_ctrl_put(struct nvme_fc_ctrl *);
@@ -2464,6 +2463,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 
 	++ctrl->ctrl.nr_reconnects;
 
+	if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+		return -ENODEV;
+
 	/*
 	 * Create the admin queue
 	 */
@@ -2739,6 +2741,10 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 		ctrl->cnum, status);
 
 	if (nvmf_should_reconnect(&ctrl->ctrl)) {
+		/* Only schedule the reconnect if the remote port is online */
+		if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+			return;
+
 		dev_info(ctrl->ctrl.device,
 			"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
 			ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
@@ -2772,12 +2778,15 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	ret = nvme_fc_create_association(ctrl);
-	if (ret)
-		nvme_fc_reconnect_or_delete(ctrl, ret);
-	else
-		dev_info(ctrl->ctrl.device,
-			"NVME-FC{%d}: controller reset complete\n", ctrl->cnum);
+	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) {
+		ret = nvme_fc_create_association(ctrl);
+		if (ret)
+			nvme_fc_reconnect_or_delete(ctrl, ret);
+		else
+			dev_info(ctrl->ctrl.device,
+				"NVME-FC{%d}: controller reset complete\n",
+				ctrl->cnum);
+	}
 }
 
 static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:14 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:14 -0700
Subject: [PATCH v4 2/5] nvme_fc: change ctlr state assignments during
 reset/reconnect
In-Reply-To: <20171025234317.13302-1-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
Message-ID: <20171025234317.13302-3-jsmart2021@gmail.com>

Clean up some of the controller state checks and add the
RESETTING->RECONNECTING state transition.

Specifically:
- the movement of the RESETTING state change and schedule of
  reset_work to core doesn't work wiht nvme_fc_error_recovery
  setting state to RECONNECTING before attempting to reset.
  Remove the state change as the reset request does it.
- In the rare cases where an error occurs right as we're
  transitioning to LIVE, defer the controller start actions.
- In error handling on teardown of associations while
  performing initial controller creation - avoid quiesce
  calls on the admin_q. They are unneeded.
- Add the RESETTING->RECONNECTING transition in the reset
  handler.

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
Reviewed-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/fc.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 647136dedfd3..4cd97757d9f6 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -1886,13 +1886,6 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 	dev_warn(ctrl->ctrl.device,
 		"NVME-FC{%d}: resetting controller\n", ctrl->cnum);
 
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
-		dev_err(ctrl->ctrl.device,
-			"NVME-FC{%d}: error_recovery: Couldn't change state "
-			"to RECONNECTING\n", ctrl->cnum);
-		return;
-	}
-
 	nvme_reset_ctrl(&ctrl->ctrl);
 }
 
@@ -2530,11 +2523,11 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
-	WARN_ON_ONCE(!changed);
 
 	ctrl->ctrl.nr_reconnects = 0;
 
-	nvme_start_ctrl(&ctrl->ctrl);
+	if (changed)
+		nvme_start_ctrl(&ctrl->ctrl);
 
 	return 0;	/* Success */
 
@@ -2602,7 +2595,8 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 	 * use blk_mq_tagset_busy_itr() and the transport routine to
 	 * terminate the exchanges.
 	 */
-	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	if (ctrl->ctrl.state != NVME_CTRL_NEW)
+		blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 
@@ -2706,12 +2700,8 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 static void
 nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 {
-	/* If we are resetting/deleting then do nothing */
-	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING) {
-		WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW ||
-			ctrl->ctrl.state == NVME_CTRL_LIVE);
+	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING)
 		return;
-	}
 
 	dev_info(ctrl->ctrl.device,
 		"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
@@ -2740,9 +2730,17 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	int ret;
 
 	nvme_stop_ctrl(&ctrl->ctrl);
+
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
+		dev_err(ctrl->ctrl.device,
+			"NVME-FC{%d}: error_recovery: Couldn't change state "
+			"to RECONNECTING\n", ctrl->cnum);
+		return;
+	}
+
 	ret = nvme_fc_create_association(ctrl);
 	if (ret)
 		nvme_fc_reconnect_or_delete(ctrl, ret);
-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:15 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:15 -0700
Subject: [PATCH v4 3/5] nvme_fc: add a dev_loss_tmo field to the remoteport
In-Reply-To: <20171025234317.13302-1-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
Message-ID: <20171025234317.13302-4-jsmart2021@gmail.com>

Add a dev_loss_tmo value, paralleling the SCSI FC transport, for device
connectivity loss.

The transport initializes the value in the nvme_fc_register_remoteport()
call. If the value is not set, a default of 60s is set.

Add a new routine to the api, nvme_fc_set_remoteport_devloss() routine,
which allows the lldd to dynamically update the value on an existing
remoteport.

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Johannes Thumshirn <jthumshirn at suse.de>
Reviewed-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/fc.c         | 31 +++++++++++++++++++++++++++++++
 include/linux/nvme-fc-driver.h | 11 +++++++++--
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 4cd97757d9f6..0e10a24a8786 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -45,6 +45,8 @@ enum nvme_fc_queue_flags {
 
 #define NVMEFC_QUEUE_DELAY	3		/* ms units */
 
+#define NVME_FC_DEFAULT_DEV_LOSS_TMO	60	/* seconds */
+
 struct nvme_fc_queue {
 	struct nvme_fc_ctrl	*ctrl;
 	struct device		*dev;
@@ -587,6 +589,11 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	newrec->remoteport.port_id = pinfo->port_id;
 	newrec->remoteport.port_state = FC_OBJSTATE_ONLINE;
 	newrec->remoteport.port_num = idx;
+	/* a registration value of dev_loss_tmo=0 results in the default */
+	if (pinfo->dev_loss_tmo)
+		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
+	else
+		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
 
 	spin_lock_irqsave(&nvme_fc_lock, flags);
 	list_add_tail(&newrec->endp_list, &lport->endp_list);
@@ -690,6 +697,30 @@ nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport)
 }
 EXPORT_SYMBOL_GPL(nvme_fc_rescan_remoteport);
 
+int
+nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
+			u32 dev_loss_tmo)
+{
+	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+
+	spin_lock_irqsave(&rport->lock, flags);
+
+	if (portptr->port_state != FC_OBJSTATE_ONLINE) {
+		spin_unlock_irqrestore(&rport->lock, flags);
+		return -EINVAL;
+	}
+
+	/* a dev_loss_tmo of 0 (immediate) is allowed to be set */
+	rport->remoteport.dev_loss_tmo = dev_loss_tmo;
+
+	spin_unlock_irqrestore(&rport->lock, flags);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_fc_set_remoteport_devloss);
+
 
 /* *********************** FC-NVME DMA Handling **************************** */
 
diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
index 2be4db353937..496ff759f84c 100644
--- a/include/linux/nvme-fc-driver.h
+++ b/include/linux/nvme-fc-driver.h
@@ -40,6 +40,8 @@
  * @node_name: FC WWNN for the port
  * @port_name: FC WWPN for the port
  * @port_role: What NVME roles are supported (see FC_PORT_ROLE_xxx)
+ * @dev_loss_tmo: maximum delay for reconnects to an association on
+ *             this device. Used only on a remoteport.
  *
  * Initialization values for dynamic port fields:
  * @port_id:      FC N_Port_ID currently assigned the port. Upper 8 bits must
@@ -50,6 +52,7 @@ struct nvme_fc_port_info {
 	u64			port_name;
 	u32			port_role;
 	u32			port_id;
+	u32			dev_loss_tmo;
 };
 
 
@@ -200,6 +203,9 @@ enum nvme_fc_obj_state {
  *             The length of the buffer corresponds to the local_priv_sz
  *             value specified in the nvme_fc_port_template supplied by
  *             the LLDD.
+ * @dev_loss_tmo: maximum delay for reconnects to an association on
+ *             this device. To modify, lldd must call
+ *             nvme_fc_set_remoteport_devloss().
  *
  * Fields with dynamic values. Values may change base on link state. LLDD
  * may reference fields directly to change them. Initialized by the
@@ -257,10 +263,9 @@ struct nvme_fc_remote_port {
 	u32 port_role;
 	u64 node_name;
 	u64 port_name;
-
 	struct nvme_fc_local_port *localport;
-
 	void *private;
+	u32 dev_loss_tmo;
 
 	/* dynamic fields */
 	u32 port_id;
@@ -446,6 +451,8 @@ int nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *remoteport);
 
 void nvme_fc_rescan_remoteport(struct nvme_fc_remote_port *remoteport);
 
+int nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *remoteport,
+			u32 dev_loss_tmo);
 
 
 /*
-- 
2.13.1



From jsmart2021 at gmail.com  Wed Oct 25 16:43:17 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Wed, 25 Oct 2017 16:43:17 -0700
Subject: [PATCH v4 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20171025234317.13302-1-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
Message-ID: <20171025234317.13302-6-jsmart2021@gmail.com>

This patch adds the dev_loss_tmo functionality to the transport.

When a remoteport is unregistered (connectivity lost), the following
actions are taken:
- the remoteport is marked DELETED
- the time when dev_loss_tmo would expire is set in the remoteport
- all controllers on the remoteport are reset.

After a controller resets, it will stall in a RECONNECTING state
waiting for one of the following:
- the controller will continue to attempt reconnect per max_retries
  and reconnect_delay. As no remoteport connectivity, the reconnect
  attempt will immediately fail. if max reconnects has not been
  reached, a new reconnect_delay timer will be schedule. If the
  current time plus another reconnect_delay exceeds when dev_loss_tmo
  expires on the remote port, then the reconnect_delay will be
  shortend to schedule no later than when dev_loss_tmo expires.
  if max reconnect attempts are reached (e.g. ctrl_loss_tmo reached)
  or dev_loss_tmo ix exceeded without connectivity, the controller
  is deleted.
- the remoteport is re-registered prior to dev_loss_tmo expiring.
  The resume of the remoteport will immediately attempt to reconnect
  each of its suspended controllers.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 281 ++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 242 insertions(+), 39 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 78ec1d47d1fc..200c1b867c0d 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -138,6 +138,7 @@ struct nvme_fc_rport {
 	struct nvme_fc_lport		*lport;
 	spinlock_t			lock;
 	struct kref			ref;
+	unsigned long			dev_loss_end;
 } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
 
 enum nvme_fcctrl_flags {
@@ -530,6 +531,102 @@ nvme_fc_rport_get(struct nvme_fc_rport *rport)
 	return kref_get_unless_zero(&rport->ref);
 }
 
+static void
+nvme_fc_resume_controller(struct nvme_fc_ctrl *ctrl)
+{
+	switch (ctrl->ctrl.state) {
+	case NVME_CTRL_NEW:
+	case NVME_CTRL_RECONNECTING:
+		/*
+		 * As all reconnects were suppressed, schedule a
+		 * connect.
+		 */
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: connectivity re-established. "
+			"Attempting reconnect\n", ctrl->cnum);
+
+		queue_delayed_work(nvme_wq, &ctrl->connect_work, 0);
+		break;
+
+	case NVME_CTRL_RESETTING:
+		/*
+		 * Controller is already in the process of terminating the
+		 * association. No need to do anything further. The reconnect
+		 * step will naturally occur after the reset completes.
+		 */
+		break;
+
+	default:
+		/* no action to take - let it delete */
+		break;
+	}
+}
+
+static struct nvme_fc_rport *
+nvme_fc_attach_to_suspended_rport(struct nvme_fc_lport *lport,
+				struct nvme_fc_port_info *pinfo)
+{
+	struct nvme_fc_rport *rport;
+	struct nvme_fc_ctrl *ctrl;
+	unsigned long flags;
+
+	spin_lock_irqsave(&nvme_fc_lock, flags);
+
+	list_for_each_entry(rport, &lport->endp_list, endp_list) {
+		if (rport->remoteport.node_name != pinfo->node_name ||
+		    rport->remoteport.port_name != pinfo->port_name)
+			continue;
+
+		if (!nvme_fc_rport_get(rport)) {
+			rport = ERR_PTR(-ENOLCK);
+			goto out_done;
+		}
+
+		spin_unlock_irqrestore(&nvme_fc_lock, flags);
+
+		spin_lock_irqsave(&rport->lock, flags);
+
+		/* has it been unregistered */
+		if (rport->remoteport.port_state != FC_OBJSTATE_DELETED) {
+			/* means lldd called us twice */
+			spin_unlock_irqrestore(&rport->lock, flags);
+			nvme_fc_rport_put(rport);
+			return ERR_PTR(-ESTALE);
+		}
+
+		rport->remoteport.port_state = FC_OBJSTATE_ONLINE;
+		rport->dev_loss_end = 0;
+
+		/*
+		 * kick off a reconnect attempt on all associations to the
+		 * remote port. A successful reconnects will resume i/o.
+		 */
+		list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
+			nvme_fc_resume_controller(ctrl);
+
+		spin_unlock_irqrestore(&rport->lock, flags);
+
+		return rport;
+	}
+
+	rport = NULL;
+
+out_done:
+	spin_unlock_irqrestore(&nvme_fc_lock, flags);
+
+	return rport;
+}
+
+static inline void
+__nvme_fc_set_dev_loss_tmo(struct nvme_fc_rport *rport,
+			struct nvme_fc_port_info *pinfo)
+{
+	if (pinfo->dev_loss_tmo)
+		rport->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
+	else
+		rport->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
+}
+
 /**
  * nvme_fc_register_remoteport - transport entry point called by an
  *                              LLDD to register the existence of a NVME
@@ -556,22 +653,45 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	unsigned long flags;
 	int ret, idx;
 
+	if (!nvme_fc_lport_get(lport)) {
+		ret = -ESHUTDOWN;
+		goto out_reghost_failed;
+	}
+
+	/*
+	 * look to see if there is already a remoteport that is waiting
+	 * for a reconnect (within dev_loss_tmo) with the same WWN's.
+	 * If so, transition to it and reconnect.
+	 */
+	newrec = nvme_fc_attach_to_suspended_rport(lport, pinfo);
+
+	/* found an rport, but something about its state is bad */
+	if (IS_ERR(newrec)) {
+		ret = PTR_ERR(newrec);
+		goto out_lport_put;
+
+	/* found existing rport, which was resumed */
+	} else if (newrec) {
+		nvme_fc_lport_put(lport);
+		__nvme_fc_set_dev_loss_tmo(newrec, pinfo);
+		nvme_fc_signal_discovery_scan(lport, newrec);
+		*portptr = &newrec->remoteport;
+		return 0;
+	}
+
+	/* nothing found - allocate a new remoteport struct */
+
 	newrec = kmalloc((sizeof(*newrec) + lport->ops->remote_priv_sz),
 			 GFP_KERNEL);
 	if (!newrec) {
 		ret = -ENOMEM;
-		goto out_reghost_failed;
-	}
-
-	if (!nvme_fc_lport_get(lport)) {
-		ret = -ESHUTDOWN;
-		goto out_kfree_rport;
+		goto out_lport_put;
 	}
 
 	idx = ida_simple_get(&lport->endp_cnt, 0, 0, GFP_KERNEL);
 	if (idx < 0) {
 		ret = -ENOSPC;
-		goto out_lport_put;
+		goto out_kfree_rport;
 	}
 
 	INIT_LIST_HEAD(&newrec->endp_list);
@@ -589,11 +709,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	newrec->remoteport.port_id = pinfo->port_id;
 	newrec->remoteport.port_state = FC_OBJSTATE_ONLINE;
 	newrec->remoteport.port_num = idx;
-	/* a registration value of dev_loss_tmo=0 results in the default */
-	if (pinfo->dev_loss_tmo)
-		newrec->remoteport.dev_loss_tmo = pinfo->dev_loss_tmo;
-	else
-		newrec->remoteport.dev_loss_tmo = NVME_FC_DEFAULT_DEV_LOSS_TMO;
+	__nvme_fc_set_dev_loss_tmo(newrec, pinfo);
 
 	spin_lock_irqsave(&nvme_fc_lock, flags);
 	list_add_tail(&newrec->endp_list, &lport->endp_list);
@@ -604,10 +720,10 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	*portptr = &newrec->remoteport;
 	return 0;
 
-out_lport_put:
-	nvme_fc_lport_put(lport);
 out_kfree_rport:
 	kfree(newrec);
+out_lport_put:
+	nvme_fc_lport_put(lport);
 out_reghost_failed:
 	*portptr = NULL;
 	return ret;
@@ -638,6 +754,61 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
 	return 0;
 }
 
+static void
+nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
+{
+	dev_info(ctrl->ctrl.device,
+		"NVME-FC{%d}: controller connectivity lost. Awaiting "
+		"Reconnect", ctrl->cnum);
+
+	switch (ctrl->ctrl.state) {
+	case NVME_CTRL_NEW:
+	case NVME_CTRL_LIVE:
+		/*
+		 * Schedule a controller reset. The reset will
+		 * terminate the association and schedule the
+		 * reconnect timer. Reconnects will be attempted
+		 * until either the ctlr_loss_tmo
+		 * (max_retries * connect_delay) expires or the
+		 * remoteport's dev_loss_tmo expires.
+		 */
+		if (nvme_reset_ctrl(&ctrl->ctrl)) {
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: Couldn't schedule reset. "
+				"Deleting controller.\n",
+				ctrl->cnum);
+			__nvme_fc_del_ctrl(ctrl);
+		}
+		break;
+
+	case NVME_CTRL_RECONNECTING:
+		/*
+		 * The association has already been terminated
+		 * and the controller is attempting reconnects.
+		 * No need to do anything futher. Reconnects will
+		 * be attempted until either the ctlr_loss_tmo
+		 * (max_retries * connect_delay) expires or the
+		 * remoteport's dev_loss_tmo expires.
+		 */
+		break;
+
+	case NVME_CTRL_RESETTING:
+		/*
+		 * Controller is already in the process of
+		 * terminating the association. No need to do
+		 * anything further. The reconnect step will
+		 * kick in naturally after the association is
+		 * terminated.
+		 */
+		break;
+
+	case NVME_CTRL_DELETING:
+	default:
+		/* no action to take - let it delete */
+		break;
+	}
+}
+
 /**
  * nvme_fc_unregister_remoteport - transport entry point called by an
  *                              LLDD to deregister/remove a previously
@@ -667,15 +838,31 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
 	}
 	portptr->port_state = FC_OBJSTATE_DELETED;
 
-	/* tear down all associations to the remote port */
-	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
-		__nvme_fc_del_ctrl(ctrl);
+	rport->dev_loss_end = jiffies + (portptr->dev_loss_tmo * HZ);
+
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
+		/* if dev_loss_tmo==0, dev loss is immediate */
+		if (!portptr->dev_loss_tmo) {
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: controller connectivity lost. "
+				"Deleting controller.\n",
+				ctrl->cnum);
+			__nvme_fc_del_ctrl(ctrl);
+		} else
+			nvme_fc_ctrl_connectivity_loss(ctrl);
+	}
 
 	spin_unlock_irqrestore(&rport->lock, flags);
 
 	nvme_fc_abort_lsops(rport);
 
+	/*
+	 * release the reference, which will allow, if all controllers
+	 * go away, which should only occur after dev_loss_tmo occurs,
+	 * for the rport to be torn down.
+	 */
 	nvme_fc_rport_put(rport);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_fc_unregister_remoteport);
@@ -702,7 +889,6 @@ nvme_fc_set_remoteport_devloss(struct nvme_fc_remote_port *portptr,
 			u32 dev_loss_tmo)
 {
 	struct nvme_fc_rport *rport = remoteport_to_rport(portptr);
-	struct nvme_fc_ctrl *ctrl;
 	unsigned long flags;
 
 	spin_lock_irqsave(&rport->lock, flags);
@@ -2733,28 +2919,43 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 static void
 nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 {
+	struct nvme_fc_rport *rport = ctrl->rport;
+	struct nvme_fc_remote_port *portptr = &rport->remoteport;
+	unsigned long recon_delay = ctrl->ctrl.opts->reconnect_delay * HZ;
+	bool recon = true;
+
 	if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING)
 		return;
 
-	dev_info(ctrl->ctrl.device,
-		"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
-		ctrl->cnum, status);
+	if (portptr->port_state == FC_OBJSTATE_ONLINE)
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
+			ctrl->cnum, status);
+	else if (time_after_eq(jiffies, rport->dev_loss_end))
+		recon = false;
 
-	if (nvmf_should_reconnect(&ctrl->ctrl)) {
-		/* Only schedule the reconnect if the remote port is online */
-		if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
-			return;
+	if (recon && nvmf_should_reconnect(&ctrl->ctrl)) {
+		if (portptr->port_state == FC_OBJSTATE_ONLINE)
+			dev_info(ctrl->ctrl.device,
+				"NVME-FC{%d}: Reconnect attempt in %ld "
+				"seconds\n",
+				ctrl->cnum, recon_delay / HZ);
+		else if (time_after(jiffies + recon_delay, rport->dev_loss_end))
+			recon_delay = rport->dev_loss_end - jiffies;
 
-		dev_info(ctrl->ctrl.device,
-			"NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
-			ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
-		queue_delayed_work(nvme_wq, &ctrl->connect_work,
-				ctrl->ctrl.opts->reconnect_delay * HZ);
+		queue_delayed_work(nvme_wq, &ctrl->connect_work, recon_delay);
 	} else {
-		dev_warn(ctrl->ctrl.device,
+		if (portptr->port_state == FC_OBJSTATE_ONLINE)
+			dev_warn(ctrl->ctrl.device,
 				"NVME-FC{%d}: Max reconnect attempts (%d) "
 				"reached. Removing controller\n",
 				ctrl->cnum, ctrl->ctrl.nr_reconnects);
+		else
+			dev_warn(ctrl->ctrl.device,
+				"NVME-FC{%d}: dev_loss_tmo (%d) expired "
+				"while waiting for remoteport connectivity. "
+				"Removing controller\n", ctrl->cnum,
+				portptr->dev_loss_tmo);
 		WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
 	}
 }
@@ -2778,15 +2979,17 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) {
+	if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE)
 		ret = nvme_fc_create_association(ctrl);
-		if (ret)
-			nvme_fc_reconnect_or_delete(ctrl, ret);
-		else
-			dev_info(ctrl->ctrl.device,
-				"NVME-FC{%d}: controller reset complete\n",
-				ctrl->cnum);
-	}
+	else
+		ret = -ENOTCONN;
+
+	if (ret)
+		nvme_fc_reconnect_or_delete(ctrl, ret);
+	else
+		dev_info(ctrl->ctrl.device,
+			"NVME-FC{%d}: controller reset complete\n",
+			ctrl->cnum);
 }
 
 static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
-- 
2.13.1



From hch at infradead.org  Thu Oct 26 23:11:27 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 26 Oct 2017 23:11:27 -0700
Subject: [PATCH 0/3] centralize transport not ready request check
In-Reply-To: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
Message-ID: <20171027061127.GA30853@infradead.org>

I like this a lot, but the big question is how we get this in as
we have a 4.14 change in this area.  Either we'll have a self-conflict
for 4.15, or we jus tdelay this to be pulled into 4.15 just after
-rc1?


From hch at infradead.org  Thu Oct 26 23:11:43 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 26 Oct 2017 23:11:43 -0700
Subject: [PATCH] nvme_fc: remove NVME_FC_MAX_SEGMENTS
In-Reply-To: <20171023221136.23216-1-jsmart2021@gmail.com>
References: <20171023221136.23216-1-jsmart2021@gmail.com>
Message-ID: <20171027061143.GB30853@infradead.org>

Thanks, applied to nvme-4.15.


From hch at infradead.org  Thu Oct 26 23:14:40 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 26 Oct 2017 23:14:40 -0700
Subject: [PATCH v2] nvme-cli: add duplicate_connect parameter support
In-Reply-To: <20171020231815.3288-1-jsmart2021@gmail.com>
References: <20171020231815.3288-1-jsmart2021@gmail.com>
Message-ID: <20171027061440.GC30853@infradead.org>

Thanks, applied the whole series to nvme-4.15.

Although I'm travelling at the moment and could not rest the RDMA part.
It would be great if someone could give it a spin before the end of
the merge window.


From hch at infradead.org  Thu Oct 26 23:23:19 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 27 Oct 2017 09:23:19 +0300
Subject: [GIT PULL] one nvme fix for 4.14
Message-ID: <20171027062319.gvxuxxffwcfh2v4q@infradead.org>

The following changes since commit f04b9cc87b5fc466b1b7231ba7b078e885956c5b:

  nvme-rdma: Fix error status return in tagset allocation failure (2017-10-19 17:13:51 +0200)

are available in the git repository at:

  git://git.infradead.org/nvme.git nvme-4.14

for you to fetch changes up to 7db814465395f3196ee98c8bd40d214d63e4f708:

  nvme-rdma: fix possible hang when issuing commands during ctrl removal (2017-10-23 16:27:44 +0200)

----------------------------------------------------------------
Sagi Grimberg (1):
      nvme-rdma: fix possible hang when issuing commands during ctrl removal

 drivers/nvme/host/rdma.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)


From hch at infradead.org  Thu Oct 26 23:23:52 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 26 Oct 2017 23:23:52 -0700
Subject: [PATCH] nvme: Remove unused headers
In-Reply-To: <20171020221803.30484-1-keith.busch@intel.com>
References: <20171020221803.30484-1-keith.busch@intel.com>
Message-ID: <20171027062352.GA25922@infradead.org>

Thanks, applied to nvme-4.15.


From hare at suse.de  Fri Oct 27 00:16:06 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 27 Oct 2017 09:16:06 +0200
Subject: [PATCH v4 5/5] nvme_fc: add dev_loss_tmo timeout and remoteport
 resume support
In-Reply-To: <20171025234317.13302-6-jsmart2021@gmail.com>
References: <20171025234317.13302-1-jsmart2021@gmail.com>
 <20171025234317.13302-6-jsmart2021@gmail.com>
Message-ID: <d79f9cf9-a910-4dfb-3cf1-d26a6c0326a5@suse.de>

On 10/26/2017 01:43 AM, James Smart wrote:
> This patch adds the dev_loss_tmo functionality to the transport.
> 
> When a remoteport is unregistered (connectivity lost), the following
> actions are taken:
> - the remoteport is marked DELETED
> - the time when dev_loss_tmo would expire is set in the remoteport
> - all controllers on the remoteport are reset.
> 
> After a controller resets, it will stall in a RECONNECTING state
> waiting for one of the following:
> - the controller will continue to attempt reconnect per max_retries
>   and reconnect_delay. As no remoteport connectivity, the reconnect
>   attempt will immediately fail. if max reconnects has not been
>   reached, a new reconnect_delay timer will be schedule. If the
>   current time plus another reconnect_delay exceeds when dev_loss_tmo
>   expires on the remote port, then the reconnect_delay will be
>   shortend to schedule no later than when dev_loss_tmo expires.
>   if max reconnect attempts are reached (e.g. ctrl_loss_tmo reached)
>   or dev_loss_tmo ix exceeded without connectivity, the controller
>   is deleted.
> - the remoteport is re-registered prior to dev_loss_tmo expiring.
>   The resume of the remoteport will immediately attempt to reconnect
>   each of its suspended controllers.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> ---
>  drivers/nvme/host/fc.c | 281 ++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 242 insertions(+), 39 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Fri Oct 27 00:16:17 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 27 Oct 2017 09:16:17 +0200
Subject: [PATCH] nvme-fc: Zero out SGL before submitting AEN
Message-ID: <1509088577-7342-1-git-send-email-hare@suse.de>

nvme_fc_start_fc_op() will be updating the SGL entries of the
sqe, so we need to zero those before resubmitting a received
AEN. Otherwise we trigger the WARN_ON() in nvme_fc_start_fc_op().

Signed-off-by: Hannes Reinecke <hare at suse.com>
---
 drivers/nvme/host/fc.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index b600c07..db0a7f6 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2181,6 +2181,7 @@ enum {
 {
 	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(arg);
 	struct nvme_fc_fcp_op *aen_op;
+	struct nvme_command *sqe;
 	unsigned long flags;
 	bool terminating = false;
 	blk_status_t ret;
@@ -2197,6 +2198,8 @@ enum {
 		return;
 
 	aen_op = &ctrl->aen_ops[aer_idx];
+	sqe = &aen_op->cmd_iu.sqe;
+	memset(&sqe->common.dptr, 0, sizeof(union nvme_data_ptr));
 
 	ret = nvme_fc_start_fcp_op(ctrl, aen_op->queue, aen_op, 0,
 					NVMEFC_FCP_NODATA);
-- 
1.8.5.6



From hare at suse.de  Fri Oct 27 00:48:16 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 27 Oct 2017 09:48:16 +0200
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <20171009201032.2018-1-jsmart2021@gmail.com>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
Message-ID: <e542bb4d-946f-a080-5701-f8019fdbc752@suse.de>

On 10/09/2017 10:10 PM, James Smart wrote:
> Sagi recently posted a "one-shot" call to nvme-connect via a systemd
> script. FC has no problems operating with it assuming all the required
> connect fields are present (host_traddr required).
> 
> However FC is a bit more dynamic, and we've built up scripts that can
> be triggered whenever a new FC remote port with a discovery subsytem is
> detected. Whenever such a port is detected (by registration of the
> remoteport with the nvme_fc transport by the lldd), the transport
> generates a udev event.  A udev rule is created that parses the event
> information into the FC remote port (traddr) that supports a discovery
> subsystem and the FC host port that is connected to the FC remote port
> (host_traddr). The rule then invokes an instance-based systemd service
> to perform a "nvme connect-all" with the traddr and host-traddr info.
> 
> To resolve some issues seen by discovery of remote ports before the udev
> service was running and where the events were not replayed, there is an
> additional systemd service that runs once and invokes a script which
> locates the FC host ports on the system, looks for any connected FC
> remote ports with discovery subsystems, and invokes a connect-all for
> them.  The FC port information is specific to lpfc attributes currently,
> but when the shared transport is in place, they will be common.
> 
> I don't claim to be a systemd expert, so comments are certainly welcome.
> 
In general I'm not in favour of calling 'systemctl start' from a udev
rule; this typically will cause systemd to rearrange the internal
dependency chain, and might trigger various services to be enabled or
disabled.

The 'normal' flow of events is:

kernel device -> uevent -> udev rules -> systemd services

ie for the autoconnect it would be _far_ better if we could have an
event per NVMe lport device, as then we could call 'connect' on the
lport device.

So a better workflow would be to have the 'discovery' event trigger a
'change' (or 'add'?) event on the lports, and then we could have
per-lport udev rules triggering the actualy connect.

Let's see if I can come up with something along these lines.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From hare at suse.de  Fri Oct 27 01:43:51 2017
From: hare at suse.de (Hannes Reinecke)
Date: Fri, 27 Oct 2017 10:43:51 +0200
Subject: [REPOST PATCH v2] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171025144130.8833-1-jsmart2021@gmail.com>
References: <20171025144130.8833-1-jsmart2021@gmail.com>
Message-ID: <d995811b-f88d-f1fe-eb25-13b380c20ae3@suse.de>

On 10/25/2017 04:41 PM, James Smart wrote:
> Below is  a stack trace for an issue that was reported.
> 
> What's happening is that the nvmet layer had it's controller kato
> timeout fire, which causes it to schedule its fatal error handler
> via the fatal_err_work element. The error handler is invoked, which
> calls the transport delete_ctrl() entry point, and as the transport
> tears down the controller, nvmet_sq_destroy ends up doing the final
> put on the ctlr causing it to enter its free routine. The ctlr free
> routine does a cancel_work_sync() on fatal_err_work element, which
> then does a flush_work and wait_for_completion. But, as the wait is
> in the context of the work element being flushed, its in a catch-22
> and the thread hangs.
> 
> [  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
> [  326.909832] nvmet: ctrl 1 fatal error occurred!
> [  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
> x114 flg x2
> [  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
> seconds.
> [  494.589638]       Not tainted 4.14.0-rc1.James+ #1
> [  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  494.603718] kworker/0:2     D    0   243      2 0x80000000
> [  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
> [  494.616447] Call Trace:
> [  494.619177]  __schedule+0x28d/0x890
> [  494.623070]  schedule+0x36/0x80
> [  494.626571]  schedule_timeout+0x1dd/0x300
> [  494.631044]  ? dequeue_task_fair+0x592/0x840
> [  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
> [  494.640756]  wait_for_completion+0x121/0x180
> [  494.645521]  ? wake_up_q+0x80/0x80
> [  494.649315]  flush_work+0x11d/0x1a0
> [  494.653206]  ? wake_up_worker+0x30/0x30
> [  494.657484]  __cancel_work_timer+0x10b/0x190
> [  494.662249]  cancel_work_sync+0x10/0x20
> [  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
> [  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
> [  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
> [  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
> [  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
> [  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
> [  494.701589]  process_one_work+0x149/0x360
> [  494.706064]  worker_thread+0x4d/0x3c0
> [  494.710148]  kthread+0x109/0x140
> [  494.713751]  ? rescuer_thread+0x380/0x380
> [  494.718214]  ? kthread_park+0x60/0x60
> 
> Correct by creating a final free work element, which is scheduled
> by the final ctrl put routine, so that when the flush_work (was
> cancel_work_sync) is called, it cannot be in the same work_q
> element context.
> 
> Signed-off-by: James Smart <james.smart at broadcom.com>
> ---
> v2:
>   convert cancel_work_sync() to flush_work() for fatal_err_work
> v3: cancelled. v2 is correct
> 
>  drivers/nvme/target/core.c  | 28 +++++++++++++++++++---------
>  drivers/nvme/target/nvmet.h |  1 +
>  2 files changed, 20 insertions(+), 9 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From axboe at kernel.dk  Fri Oct 27 07:18:15 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Fri, 27 Oct 2017 08:18:15 -0600
Subject: [GIT PULL] one nvme fix for 4.14
In-Reply-To: <20171027062319.gvxuxxffwcfh2v4q@infradead.org>
References: <20171027062319.gvxuxxffwcfh2v4q@infradead.org>
Message-ID: <8d69e358-756b-6ab9-5c18-0671d926cbd4@kernel.dk>

On 10/27/2017 12:23 AM, Christoph Hellwig wrote:
> The following changes since commit f04b9cc87b5fc466b1b7231ba7b078e885956c5b:
> 
>   nvme-rdma: Fix error status return in tagset allocation failure (2017-10-19 17:13:51 +0200)
> 
> are available in the git repository at:
> 
>   git://git.infradead.org/nvme.git nvme-4.14
> 
> for you to fetch changes up to 7db814465395f3196ee98c8bd40d214d63e4f708:
> 
>   nvme-rdma: fix possible hang when issuing commands during ctrl removal (2017-10-23 16:27:44 +0200)
> 
> ----------------------------------------------------------------
> Sagi Grimberg (1):
>       nvme-rdma: fix possible hang when issuing commands during ctrl removal

Pulled, thanks.

-- 
Jens Axboe



From axboe at kernel.dk  Fri Oct 27 07:25:47 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Fri, 27 Oct 2017 08:25:47 -0600
Subject: [PATCH 0/3] centralize transport not ready request check
In-Reply-To: <20171027061127.GA30853@infradead.org>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
 <20171027061127.GA30853@infradead.org>
Message-ID: <7cff5a68-cec9-ef1f-201d-f98c631137b4@kernel.dk>

On 10/27/2017 12:11 AM, Christoph Hellwig wrote:
> I like this a lot, but the big question is how we get this in as
> we have a 4.14 change in this area.  Either we'll have a self-conflict
> for 4.15, or we jus tdelay this to be pulled into 4.15 just after
> -rc1?

I can just setup a post-merge branch, which will become a for-linus
branch that I'll send in during the merge window as well.

-- 
Jens Axboe



From axboe at kernel.dk  Fri Oct 27 09:35:58 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Fri, 27 Oct 2017 10:35:58 -0600
Subject: [PATCH] nvme: freeze IO accesses around format
Message-ID: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>

If someone attempts to do IO to a drive while it is under format,
we risk timing out that IO. That potentially leads to the driver
triggering a controller reset, and subsequently the format is ruined and
the device goes away.

Prevents this by freezing IO access to the device around a format.
Without this, the following set of commands can easily make your device
disappear:

parted -s /dev/nvme3n1 mklabel gpt
parted -s /dev/nvme3n1 mkpart primary 0G 100G
parted -s /dev/nvme3n1 rm 1
nvme format /dev/nvme3

since the last partition removal will trigger a udev partition reload,
which happens while the format is running. If the format takes longer
than the normal IO timeout, we start timing it out:

[  456.799438]  nvme3n1:
[  456.833656]  nvme3n1: p1
[  456.842025]  nvme3n1: p1
[  456.887368]  nvme3n1:
[  487.699023] nvme nvme3: I/O 879 QID 12 timeout, aborting
[  518.098840] nvme nvme3: I/O 879 QID 12 timeout, reset controller
[  571.700471] nvme nvme3: Abort status: 0x7
[  571.798306] nvme nvme3: Removing after probe failure status: -22
[  571.811330] nvme3n1: detected capacity change from 4000787030016 to 0
[  571.819189] print_req_error: I/O error, dev nvme3n1, sector 7814036992

and the device is gone, needing a driver reload or reboot to bring it
back. Same thing happens if you just do a dd from the device and then
start a format. Behavior is vendor agnostic, basically just timing
dependent.

Signed-off-by: Jens Axboe <axboe at kernel.dk>

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 5a14cc7f28ee..13d7fda73fbc 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1013,10 +1013,23 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
+	/*
+	 * Freeze current access to the device, and prevent new ones, around
+	 * a format operation.
+	 */
+	if (cmd.opcode == nvme_admin_format_nvm) {
+		nvme_start_freeze(ctrl);
+		nvme_wait_freeze(ctrl);
+	}
+
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			(void __user *)(uintptr_t)cmd.addr, cmd.data_len,
 			(void __user *)(uintptr_t)cmd.metadata, cmd.metadata,
 			0, &cmd.result, timeout);
+
+	if (cmd.opcode == nvme_admin_format_nvm)
+		nvme_unfreeze(ctrl);
+
 	if (status >= 0) {
 		if (put_user(cmd.result, &ucmd->result))
 			return -EFAULT;

-- 
Jens Axboe



From keith.busch at intel.com  Fri Oct 27 09:44:40 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 27 Oct 2017 10:44:40 -0600
Subject: [PATCH] nvme: freeze IO accesses around format
In-Reply-To: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
References: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
Message-ID: <20171027164440.GA8644@localhost.localdomain>

On Fri, Oct 27, 2017 at 10:35:58AM -0600, Jens Axboe wrote:
> If someone attempts to do IO to a drive while it is under format,
> we risk timing out that IO. That potentially leads to the driver
> triggering a controller reset, and subsequently the format is ruined and
> the device goes away.
> 
> Prevents this by freezing IO access to the device around a format.
> Without this, the following set of commands can easily make your device
> disappear:
> 
> parted -s /dev/nvme3n1 mklabel gpt
> parted -s /dev/nvme3n1 mkpart primary 0G 100G
> parted -s /dev/nvme3n1 rm 1
> nvme format /dev/nvme3
> 
> since the last partition removal will trigger a udev partition reload,
> which happens while the format is running. If the format takes longer
> than the normal IO timeout, we start timing it out:
> 
> [  456.799438]  nvme3n1:
> [  456.833656]  nvme3n1: p1
> [  456.842025]  nvme3n1: p1
> [  456.887368]  nvme3n1:
> [  487.699023] nvme nvme3: I/O 879 QID 12 timeout, aborting
> [  518.098840] nvme nvme3: I/O 879 QID 12 timeout, reset controller
> [  571.700471] nvme nvme3: Abort status: 0x7
> [  571.798306] nvme nvme3: Removing after probe failure status: -22
> [  571.811330] nvme3n1: detected capacity change from 4000787030016 to 0
> [  571.819189] print_req_error: I/O error, dev nvme3n1, sector 7814036992
> 
> and the device is gone, needing a driver reload or reboot to bring it
> back. Same thing happens if you just do a dd from the device and then
> start a format. Behavior is vendor agnostic, basically just timing
> dependent.
> 
> Signed-off-by: Jens Axboe <axboe at kernel.dk>

Looks good.

Reviewed-by: Keith Busch <keith.busch at intel.com>


From jsmart2021 at gmail.com  Fri Oct 27 10:28:03 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 27 Oct 2017 10:28:03 -0700
Subject: [PATCH 0/2] nvme_fc: fix unregistrations and stale resume
Message-ID: <20171027172805.13495-1-jsmart2021@gmail.com>

These patches fix some errors in the nvme_fc_register_localport resume
as well as reworking remoteport and localport unregister behavior relative
to their callbacks to the lldd.

James Smart (2):
  nvme_fc: fix localport resume using stale values
  nvme_fc: decouple ns references from lldd references

 drivers/nvme/host/fc.c | 96 +++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 88 insertions(+), 8 deletions(-)

-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 27 10:28:04 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 27 Oct 2017 10:28:04 -0700
Subject: [PATCH 1/2] nvme_fc: fix localport resume using stale values
In-Reply-To: <20171027172805.13495-1-jsmart2021@gmail.com>
References: <20171027172805.13495-1-jsmart2021@gmail.com>
Message-ID: <20171027172805.13495-2-jsmart2021@gmail.com>

The localport resume was not updating the lldd ops structure. If the
lldd is unloaded and reloaded, the ops pointers will differ.

Additionally, as there are device references taken by the localport,
ensure that resume only resumes if the device matches as well.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/host/fc.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 200c1b867c0d..e2054c90b42d 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -270,7 +270,9 @@ nvme_fc_lport_get(struct nvme_fc_lport *lport)
 
 
 static struct nvme_fc_lport *
-nvme_fc_attach_to_unreg_lport(struct nvme_fc_port_info *pinfo)
+nvme_fc_attach_to_unreg_lport(struct nvme_fc_port_info *pinfo,
+			struct nvme_fc_port_template *ops,
+			struct device *dev)
 {
 	struct nvme_fc_lport *lport;
 	unsigned long flags;
@@ -282,6 +284,11 @@ nvme_fc_attach_to_unreg_lport(struct nvme_fc_port_info *pinfo)
 		    lport->localport.port_name != pinfo->port_name)
 			continue;
 
+		if (lport->dev != dev) {
+			lport = ERR_PTR(-EXDEV);
+			goto out_done;
+		}
+
 		if (lport->localport.port_state != FC_OBJSTATE_DELETED) {
 			lport = ERR_PTR(-EEXIST);
 			goto out_done;
@@ -298,6 +305,7 @@ nvme_fc_attach_to_unreg_lport(struct nvme_fc_port_info *pinfo)
 
 		/* resume the lport */
 
+		lport->ops = ops;
 		lport->localport.port_role = pinfo->port_role;
 		lport->localport.port_id = pinfo->port_id;
 		lport->localport.port_state = FC_OBJSTATE_ONLINE;
@@ -358,7 +366,7 @@ nvme_fc_register_localport(struct nvme_fc_port_info *pinfo,
 	 * expired, we can simply re-enable the localport. Remoteports
 	 * and controller reconnections should resume naturally.
 	 */
-	newrec = nvme_fc_attach_to_unreg_lport(pinfo);
+	newrec = nvme_fc_attach_to_unreg_lport(pinfo, template, dev);
 
 	/* found an lport, but something about its state is bad */
 	if (IS_ERR(newrec)) {
-- 
2.13.1



From jsmart2021 at gmail.com  Fri Oct 27 10:28:05 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 27 Oct 2017 10:28:05 -0700
Subject: [PATCH 2/2] nvme_fc: decouple ns references from lldd references
In-Reply-To: <20171027172805.13495-1-jsmart2021@gmail.com>
References: <20171027172805.13495-1-jsmart2021@gmail.com>
Message-ID: <20171027172805.13495-3-jsmart2021@gmail.com>

In the lldd api, a lldd may unregister a remoteport (loss of connectivity
or driver unload) or localport (driver unload). The lldd must wait for the
remoteport_delete or localport_delete before completing its actions post
the unregister.  The xxx_deletes currently occur only when the xxxport
structure is fully freed after all references are removed. Thus the lldd
may be held hostage until an app or in-kernel entity that has a namespace
open finally closes so the namespace can be removed, the controller
removed, thus the transport objects, thus the lldd.

This patch decouples the transport and os-facing objects from the lldd
and the remoteport and localport. There is a point in all deletions
where the transport will no longer interact with the lldd on behalf of
a controller. That point centers around the association established
with the target/subsystem. It will access the lldd whenever it attempts
to create an association and while the association is active. New
associations may only be created if the remoteport is live (thus the
localport is live). It will not access the lldd after deleting the
association.

Therefore, the patch tracks the count of active controllers - those with
associations being created or that are active - on a remoteport. It also
tracks the number of remoteports that have active controllers, on a
a localport. When a remoteport is unregistered, as soon as there are no
active controllers, the lldd's remoteport_delete may be called and the
lldd may continue. Similarly, when a localport is unregistered, as soon
as there are no remoteports with active controllers, the localport_delete
callback may be made. This significantly speeds up unregistration with
the lldd.

The transport objects continue in suspended status with reconnect timers
running, and upon expiration, normal ref-counting will occur and the
objects will be freed. The transport object may still be held hostage
by the application/kernel module, but that is acceptable.

With this change, the lldd may be fully unloaded and reloaded, and
if registrations occur prior to the timeouts, the nvme controller and
namespaces will resume normally as if a link bounce.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
NOTE: this patch was cut against nvme-4.15, but with the nvme_fc
devloss patches also applied prior. This is meaningful as this patch
requires the
"nvme_fc: check connectivity before initiating reconnects" patch
http://lists.infradead.org/pipermail/linux-nvme/2017-October/013664.html
which has this change, which is critical:
 @@ -2464,6 +2463,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)

     	++ctrl->ctrl.nr_reconnects;

 +	if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
 +		return -ENODEV;
 +
  	/*
 	 * Create the admin queue
  	 */

 drivers/nvme/host/fc.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 78 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index e2054c90b42d..9b0c6eeb6f37 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -126,6 +126,7 @@ struct nvme_fc_lport {
 	struct device			*dev;	/* physical device for dma */
 	struct nvme_fc_port_template	*ops;
 	struct kref			ref;
+	atomic_t                        act_rport_cnt;
 } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
 
 struct nvme_fc_rport {
@@ -138,6 +139,7 @@ struct nvme_fc_rport {
 	struct nvme_fc_lport		*lport;
 	spinlock_t			lock;
 	struct kref			ref;
+	atomic_t                        act_ctrl_cnt;
 	unsigned long			dev_loss_end;
 } __aligned(sizeof(u64));	/* alignment for other things alloc'd with */
 
@@ -153,6 +155,7 @@ struct nvme_fc_ctrl {
 	struct nvme_fc_rport	*rport;
 	u32			cnum;
 
+	bool			assoc_active;
 	u64			association_id;
 
 	struct list_head	ctrl_list;	/* rport->ctrl_list */
@@ -245,9 +248,6 @@ nvme_fc_free_lport(struct kref *ref)
 	list_del(&lport->port_list);
 	spin_unlock_irqrestore(&nvme_fc_lock, flags);
 
-	/* let the LLDD know we've finished tearing it down */
-	lport->ops->localport_delete(&lport->localport);
-
 	ida_simple_remove(&nvme_fc_local_port_cnt, lport->localport.port_num);
 	ida_destroy(&lport->endp_cnt);
 
@@ -402,6 +402,7 @@ nvme_fc_register_localport(struct nvme_fc_port_info *pinfo,
 	INIT_LIST_HEAD(&newrec->port_list);
 	INIT_LIST_HEAD(&newrec->endp_list);
 	kref_init(&newrec->ref);
+	atomic_set(&newrec->act_rport_cnt, 0);
 	newrec->ops = template;
 	newrec->dev = dev;
 	ida_init(&newrec->endp_cnt);
@@ -464,6 +465,9 @@ nvme_fc_unregister_localport(struct nvme_fc_local_port *portptr)
 
 	spin_unlock_irqrestore(&nvme_fc_lock, flags);
 
+	if (atomic_read(&lport->act_rport_cnt) == 0)
+		lport->ops->localport_delete(&lport->localport);
+
 	nvme_fc_lport_put(lport);
 
 	return 0;
@@ -517,9 +521,6 @@ nvme_fc_free_rport(struct kref *ref)
 	list_del(&rport->endp_list);
 	spin_unlock_irqrestore(&nvme_fc_lock, flags);
 
-	/* let the LLDD know we've finished tearing it down */
-	lport->ops->remoteport_delete(&rport->remoteport);
-
 	ida_simple_remove(&lport->endp_cnt, rport->remoteport.port_num);
 
 	kfree(rport);
@@ -706,6 +707,7 @@ nvme_fc_register_remoteport(struct nvme_fc_local_port *localport,
 	INIT_LIST_HEAD(&newrec->ctrl_list);
 	INIT_LIST_HEAD(&newrec->ls_req_list);
 	kref_init(&newrec->ref);
+	atomic_set(&newrec->act_ctrl_cnt, 0);
 	spin_lock_init(&newrec->lock);
 	newrec->remoteport.localport = &lport->localport;
 	newrec->dev = lport->dev;
@@ -864,6 +866,9 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
 
 	nvme_fc_abort_lsops(rport);
 
+	if (atomic_read(&rport->act_ctrl_cnt) == 0)
+		rport->lport->ops->remoteport_delete(portptr);
+
 	/*
 	 * release the reference, which will allow, if all controllers
 	 * go away, which should only occur after dev_loss_tmo occurs,
@@ -2644,6 +2649,61 @@ nvme_fc_reinit_io_queues(struct nvme_fc_ctrl *ctrl)
 	return ret;
 }
 
+static void
+nvme_fc_rport_active_on_lport(struct nvme_fc_rport *rport)
+{
+	struct nvme_fc_lport *lport = rport->lport;
+
+	atomic_inc(&lport->act_rport_cnt);
+}
+
+static void
+nvme_fc_rport_inactive_on_lport(struct nvme_fc_rport *rport)
+{
+	struct nvme_fc_lport *lport = rport->lport;
+	u32 cnt;
+
+	cnt = atomic_dec_return(&lport->act_rport_cnt);
+	if (cnt == 0 && lport->localport.port_state == FC_OBJSTATE_DELETED)
+		lport->ops->localport_delete(&lport->localport);
+}
+
+static int
+nvme_fc_ctlr_active_on_rport(struct nvme_fc_ctrl *ctrl)
+{
+	struct nvme_fc_rport *rport = ctrl->rport;
+	u32 cnt;
+
+	if (ctrl->assoc_active)
+		return 1;
+
+	ctrl->assoc_active = true;
+	cnt = atomic_inc_return(&rport->act_ctrl_cnt);
+	if (cnt == 1)
+		nvme_fc_rport_active_on_lport(rport);
+
+	return 0;
+}
+
+static int
+nvme_fc_ctlr_inactive_on_rport(struct nvme_fc_ctrl *ctrl)
+{
+	struct nvme_fc_rport *rport = ctrl->rport;
+	struct nvme_fc_lport *lport = rport->lport;
+	u32 cnt;
+
+	/* ctrl->assoc_active=false will be set independently */
+
+	cnt = atomic_dec_return(&rport->act_ctrl_cnt);
+	if (cnt == 0) {
+		if (rport->remoteport.port_state == FC_OBJSTATE_DELETED)
+			lport->ops->remoteport_delete(&rport->remoteport);
+		nvme_fc_rport_inactive_on_lport(rport);
+	}
+
+	return 0;
+}
+
 /*
  * This routine restarts the controller on the host side, and
  * on the link side, recreates the controller association.
@@ -2660,6 +2720,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
 		return -ENODEV;
 
+	if (nvme_fc_ctlr_active_on_rport(ctrl))
+		return -ENOTUNIQ;
+
 	/*
 	 * Create the admin queue
 	 */
@@ -2767,6 +2830,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	__nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
 out_free_queue:
 	nvme_fc_free_queue(&ctrl->queues[0]);
+	ctrl->assoc_active = false;
+	nvme_fc_ctlr_inactive_on_rport(ctrl);
 
 	return ret;
 }
@@ -2782,6 +2847,10 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 {
 	unsigned long flags;
 
+	if (!ctrl->assoc_active)
+		return;
+	ctrl->assoc_active = false;
+
 	spin_lock_irqsave(&ctrl->lock, flags);
 	ctrl->flags |= FCCTRL_TERMIO;
 	ctrl->iocnt = 0;
@@ -2854,6 +2923,8 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 
 	__nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
 	nvme_fc_free_queue(&ctrl->queues[0]);
+
+	nvme_fc_ctlr_inactive_on_rport(ctrl);
 }
 
 static void
@@ -3109,6 +3180,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	ctrl->dev = lport->dev;
 	ctrl->cnum = idx;
 	init_waitqueue_head(&ctrl->ioabort_wait);
+	ctrl->assoc_active = false;
 
 	get_device(ctrl->dev);
 	kref_init(&ctrl->ref);
-- 
2.13.1



From james.smart at broadcom.com  Fri Oct 27 10:47:16 2017
From: james.smart at broadcom.com (James Smart)
Date: Fri, 27 Oct 2017 10:47:16 -0700
Subject: [PATCH] nvme-fc: Zero out SGL before submitting AEN
In-Reply-To: <1509088577-7342-1-git-send-email-hare@suse.de>
References: <1509088577-7342-1-git-send-email-hare@suse.de>
Message-ID: <c1e37e1d-e5d4-c939-9569-36999c0fdaed@broadcom.com>

On 10/27/2017 12:16 AM, Hannes Reinecke wrote:
> nvme_fc_start_fc_op() will be updating the SGL entries of the
> sqe, so we need to zero those before resubmitting a received
> AEN. Otherwise we trigger the WARN_ON() in nvme_fc_start_fc_op().
>
> Signed-off-by: Hannes Reinecke <hare at suse.com>
> ---
>
This patch is unnecessary. It was completely resolved by the following 
patch which has been committed:

nvme-fc: use transport-specific sgl format
http://lists.infradead.org/pipermail/linux-nvme/2017-September/012723.html

-- james



From james.smart at broadcom.com  Fri Oct 27 10:50:06 2017
From: james.smart at broadcom.com (James Smart)
Date: Fri, 27 Oct 2017 10:50:06 -0700
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <e542bb4d-946f-a080-5701-f8019fdbc752@suse.de>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
 <e542bb4d-946f-a080-5701-f8019fdbc752@suse.de>
Message-ID: <352f980c-2422-f917-a551-99bedf27e0ab@broadcom.com>

On 10/27/2017 12:48 AM, Hannes Reinecke wrote:
> On 10/09/2017 10:10 PM, James Smart wrote:
>> Sagi recently posted a "one-shot" call to nvme-connect via a systemd
>> script. FC has no problems operating with it assuming all the required
>> connect fields are present (host_traddr required).
>>
>> However FC is a bit more dynamic, and we've built up scripts that can
>> be triggered whenever a new FC remote port with a discovery subsytem is
>> detected. Whenever such a port is detected (by registration of the
>> remoteport with the nvme_fc transport by the lldd), the transport
>> generates a udev event.  A udev rule is created that parses the event
>> information into the FC remote port (traddr) that supports a discovery
>> subsystem and the FC host port that is connected to the FC remote port
>> (host_traddr). The rule then invokes an instance-based systemd service
>> to perform a "nvme connect-all" with the traddr and host-traddr info.
>>
>> To resolve some issues seen by discovery of remote ports before the udev
>> service was running and where the events were not replayed, there is an
>> additional systemd service that runs once and invokes a script which
>> locates the FC host ports on the system, looks for any connected FC
>> remote ports with discovery subsystems, and invokes a connect-all for
>> them.  The FC port information is specific to lpfc attributes currently,
>> but when the shared transport is in place, they will be common.
>>
>> I don't claim to be a systemd expert, so comments are certainly welcome.
>>
> In general I'm not in favour of calling 'systemctl start' from a udev
> rule; this typically will cause systemd to rearrange the internal
> dependency chain, and might trigger various services to be enabled or
> disabled.
>
> The 'normal' flow of events is:
>
> kernel device -> uevent -> udev rules -> systemd services
>
> ie for the autoconnect it would be _far_ better if we could have an
> event per NVMe lport device, as then we could call 'connect' on the
> lport device.
>
> So a better workflow would be to have the 'discovery' event trigger a
> 'change' (or 'add'?) event on the lports, and then we could have
> per-lport udev rules triggering the actualy connect.
>
> Let's see if I can come up with something along these lines.
>
> Cheers,
>
> Hannes

I'm certainly open to better options. udev/systemd is far from nice.? 
However, what I did has quite a bit of precedence.

I have no idea what you would scan on by just an lport. It's not enough 
for nvme-cli. You must have the lport and the rport information, which 
is passed via the udev event info and which must also be passed to nvme 
connect-all

-- james



From keith.busch at intel.com  Fri Oct 27 12:51:22 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 27 Oct 2017 13:51:22 -0600
Subject: [PATCH urgent-4.14] nvme: Fix setting logical block format when
 revalidating
Message-ID: <20171027195122.1810-1-keith.busch@intel.com>

Revalidating the disk needs to set the logical block format and capacity,
otherwise it can't figure out if the users modified anything about
the namespace.

Fixes: cdbff4f26bd9 ("nvme: remove nvme_revalidate_ns")

Signed-off-by: Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 5a14cc7f28ee..37f9039bb9ca 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1249,6 +1249,7 @@ static int nvme_revalidate_disk(struct gendisk *disk)
 		goto out;
 	}
 
+	__nvme_revalidate_disk(disk, id);
 	nvme_report_ns_ids(ctrl, ns->ns_id, id, eui64, nguid, &uuid);
 	if (!uuid_equal(&ns->uuid, &uuid) ||
 	    memcmp(&ns->nguid, &nguid, sizeof(ns->nguid)) ||
-- 
2.13.6



From james.smart at broadcom.com  Fri Oct 27 13:09:46 2017
From: james.smart at broadcom.com (James Smart)
Date: Fri, 27 Oct 2017 13:09:46 -0700
Subject: [PATCH 2/3] nvme-fc: Check if queue is ready in queue_rq
In-Reply-To: <1508847922-26699-3-git-send-email-sagi@grimberg.me>
References: <1508847922-26699-1-git-send-email-sagi@grimberg.me>
 <1508847922-26699-3-git-send-email-sagi@grimberg.me>
Message-ID: <8283bcee-394e-af25-b6ce-bf48e218d7a2@broadcom.com>

On 10/24/2017 5:25 AM, Sagi Grimberg wrote:
> In case the queue is not LIVE (fully functional
> and connected in the nvmf level), we cannot
> allow any commands other than connect to pass
> through.
>
> Add a new queue state flag NVME_FC_Q_LIVE which is
> set after nvmf connect and cleared in queue teardown.
>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>
looks fine

Reviewed-by: James Smart <james.smart at broadcom.com>

-- james


From jsmart2021 at gmail.com  Fri Oct 27 13:12:49 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Fri, 27 Oct 2017 13:12:49 -0700
Subject: [PATCH v4] nvmet: Fix fatal_err_work deadlock
Message-ID: <20171027201249.31728-1-jsmart2021@gmail.com>

Below is  a stack trace for an issue that was reported.

What's happening is that the nvmet layer had it's controller kato
timeout fire, which causes it to schedule its fatal error handler
via the fatal_err_work element. The error handler is invoked, which
calls the transport delete_ctrl() entry point, and as the transport
tears down the controller, nvmet_sq_destroy ends up doing the final
put on the ctlr causing it to enter its free routine. The ctlr free
routine does a cancel_work_sync() on fatal_err_work element, which
then does a flush_work and wait_for_completion. But, as the wait is
in the context of the work element being flushed, its in a catch-22
and the thread hangs.

[  326.903131] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  326.909832] nvmet: ctrl 1 fatal error occurred!
[  327.643100] lpfc 0000:04:00.0: 0:6313 NVMET Defer ctx release xri
x114 flg x2
[  494.582064] INFO: task kworker/0:2:243 blocked for more than 120
seconds.
[  494.589638]       Not tainted 4.14.0-rc1.James+ #1
[  494.594986] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  494.603718] kworker/0:2     D    0   243      2 0x80000000
[  494.609839] Workqueue: events nvmet_fatal_error_handler [nvmet]
[  494.616447] Call Trace:
[  494.619177]  __schedule+0x28d/0x890
[  494.623070]  schedule+0x36/0x80
[  494.626571]  schedule_timeout+0x1dd/0x300
[  494.631044]  ? dequeue_task_fair+0x592/0x840
[  494.635810]  ? pick_next_task_fair+0x23b/0x5c0
[  494.640756]  wait_for_completion+0x121/0x180
[  494.645521]  ? wake_up_q+0x80/0x80
[  494.649315]  flush_work+0x11d/0x1a0
[  494.653206]  ? wake_up_worker+0x30/0x30
[  494.657484]  __cancel_work_timer+0x10b/0x190
[  494.662249]  cancel_work_sync+0x10/0x20
[  494.666525]  nvmet_ctrl_put+0xa3/0x100 [nvmet]
[  494.671482]  nvmet_sq_:q+0x64/0xd0 [nvmet]
[  494.676540]  nvmet_fc_delete_target_queue+0x202/0x220 [nvmet_fc]
[  494.683245]  nvmet_fc_delete_target_assoc+0x6d/0xc0 [nvmet_fc]
[  494.689743]  nvmet_fc_delete_ctrl+0x137/0x1a0 [nvmet_fc]
[  494.695673]  nvmet_fatal_error_handler+0x30/0x40 [nvmet]
[  494.701589]  process_one_work+0x149/0x360
[  494.706064]  worker_thread+0x4d/0x3c0
[  494.710148]  kthread+0x109/0x140
[  494.713751]  ? rescuer_thread+0x380/0x380
[  494.718214]  ? kthread_park+0x60/0x60

Correct by having the fc transport convert to a different workq context
for the actual controller teardown which may call the cancel_work_sync.

Signed-off-by: James Smart <james.smart at broadcom.com>

---
v4:
  walk away from changing nvmet_fatal_error_handler and context in core.
  Adapt nvmet-fc transport to switch context in the delete_ctrl callback
  like the other transports.
---
 drivers/nvme/target/fc.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c
index 58e010bdda3e..5f2570f5dfa9 100644
--- a/drivers/nvme/target/fc.c
+++ b/drivers/nvme/target/fc.c
@@ -150,6 +150,7 @@ struct nvmet_fc_tgt_assoc {
 	struct list_head		a_list;
 	struct nvmet_fc_tgt_queue	*queues[NVMET_NR_QUEUES + 1];
 	struct kref			ref;
+	struct work_struct		del_work;
 };
 
 
@@ -232,6 +233,7 @@ static void nvmet_fc_tgtport_put(struct nvmet_fc_tgtport *tgtport);
 static int nvmet_fc_tgtport_get(struct nvmet_fc_tgtport *tgtport);
 static void nvmet_fc_handle_fcp_rqst(struct nvmet_fc_tgtport *tgtport,
 					struct nvmet_fc_fcp_iod *fod);
+static void nvmet_fc_delete_target_assoc(struct nvmet_fc_tgt_assoc *assoc);
 
 
 /* *********************** FC-NVME DMA Handling **************************** */
@@ -802,6 +804,16 @@ nvmet_fc_find_target_queue(struct nvmet_fc_tgtport *tgtport,
 	return NULL;
 }
 
+static void
+nvmet_fc_delete_assoc(struct work_struct *work)
+{
+	struct nvmet_fc_tgt_assoc *assoc =
+		container_of(work, struct nvmet_fc_tgt_assoc, del_work);
+
+	nvmet_fc_delete_target_assoc(assoc);
+	nvmet_fc_tgt_a_put(assoc);
+}
+
 static struct nvmet_fc_tgt_assoc *
 nvmet_fc_alloc_target_assoc(struct nvmet_fc_tgtport *tgtport)
 {
@@ -826,6 +838,7 @@ nvmet_fc_alloc_target_assoc(struct nvmet_fc_tgtport *tgtport)
 	assoc->a_id = idx;
 	INIT_LIST_HEAD(&assoc->a_list);
 	kref_init(&assoc->ref);
+	INIT_WORK(&assoc->del_work, nvmet_fc_delete_assoc);
 
 	while (needrandom) {
 		get_random_bytes(&ran, sizeof(ran) - BYTES_FOR_QID);
@@ -1118,8 +1131,7 @@ nvmet_fc_delete_ctrl(struct nvmet_ctrl *ctrl)
 		nvmet_fc_tgtport_put(tgtport);
 
 		if (found_ctrl) {
-			nvmet_fc_delete_target_assoc(assoc);
-			nvmet_fc_tgt_a_put(assoc);
+			schedule_work(&assoc->del_work);
 			return;
 		}
 
-- 
2.13.1



From keith.busch at intel.com  Fri Oct 27 16:07:56 2017
From: keith.busch at intel.com (Keith Busch)
Date: Fri, 27 Oct 2017 17:07:56 -0600
Subject: [PATCH] nvme: freeze IO accesses around format
In-Reply-To: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
References: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
Message-ID: <20171027230756.GA17245@localhost.localdomain>

On Fri, Oct 27, 2017 at 10:35:58AM -0600, Jens Axboe wrote:
> @@ -1013,10 +1013,23 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>  	if (cmd.timeout_ms)
>  		timeout = msecs_to_jiffies(cmd.timeout_ms);
>  
> +	/*
> +	 * Freeze current access to the device, and prevent new ones, around
> +	 * a format operation.
> +	 */
> +	if (cmd.opcode == nvme_admin_format_nvm) {
> +		nvme_start_freeze(ctrl);
> +		nvme_wait_freeze(ctrl);
> +	}
> +
>  	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
>  			(void __user *)(uintptr_t)cmd.addr, cmd.data_len,
>  			(void __user *)(uintptr_t)cmd.metadata, cmd.metadata,
>  			0, &cmd.result, timeout);
> +
> +	if (cmd.opcode == nvme_admin_format_nvm)
> +		nvme_unfreeze(ctrl);
> +

Testing some obscure vendor specific IO commands, I noticed this
will deadlock if an IO command opcode happens to be the same as an
nvme_admin_format_nvm command since the IO queues are all frozen.

You can check 'ns == NULL' before checking the opcode.

I also have this patch that's more generic and will perform actions
based on the controller's command effects:

---
commit 760052c4f2d697c19e0f3a78f2f9bc36b2b6b92f
Author: Keith Busch <keith.busch at intel.com>
Date:   Fri Oct 27 10:35:58 2017 -0600

    nvme: Check admin passthru command effects
    
    This patch saves the effects log page if the controller supports it,
    and performs appropriate actions before and after an admin passthrough
    command is completed.
    
    The NVMe standard provides a command effects log page so the host is
    aware of actions it may need to do in response to a particular passthrough
    command. For example, the command may need to run with IO quiesced to
    prevent timeouts or undefined behavior, or it may change the logical
    block formats that alter how the host needs to construct commands.
    
    If the controller does not support the command effects log page, the
    driver will define the effects for known opcodes. The nvme format is
    the only such command in this patch with known effects.
    
    Signed-off-by: Keith Busch <keith.busch at intel.com>

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 37f9039bb9ca..9b352541672d 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -76,6 +76,10 @@ static DEFINE_SPINLOCK(dev_list_lock);
 
 static struct class *nvme_class;
 
+static void nvme_scan(struct nvme_ctrl *ctrl);
+static void nvme_ns_remove(struct nvme_ns *ns);
+static int nvme_revalidate_disk(struct gendisk *disk);
+
 static __le32 nvme_get_log_dw10(u8 lid, size_t size)
 {
 	return cpu_to_le32((((size / 4) - 1) << 16) | lid);
@@ -982,12 +986,68 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, io.slba, NULL, 0);
 }
 
+static void nvme_get_admin_effects(struct nvme_ctrl *ctrl, u8 opcode, u32 *effects)
+{
+	if (ctrl->effects)
+		*effects = le32_to_cpu(ctrl->effects->acs[opcode]);
+
+	if (*effects != 0)
+		return;
+
+	/*
+	 * Set known effects for opcodes if the controller doesn't support
+	 * reporting the command's effects.
+	 */
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		*effects = NVME_CMD_FX_CSUPP | NVME_CMD_FX_LBCC |
+					NVME_CMD_FX_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+}
+
+static void nvme_passthru_start(struct nvme_ctrl *ctrl, u32 effects)
+{
+	if (effects & (NVME_CMD_FX_LBCC | NVME_CMD_FX_CSE_MASK)) {
+		nvme_start_freeze(ctrl);
+		nvme_wait_freeze(ctrl);
+	}
+}
+
+static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
+{
+	/*
+	 * Revaildate LBA changes prior to unfreezing. This is necessary to
+	 * prevent memory corruption if a logical block size was changed by
+	 * this command.
+	 */
+	if (effects & NVME_CMD_FX_LBCC) {
+		struct nvme_ns *ns;
+
+		mutex_lock(&ctrl->namespaces_mutex);
+		list_for_each_entry(ns, &ctrl->namespaces, list) {
+			if (ns->disk && nvme_revalidate_disk(ns->disk))
+				nvme_ns_remove(ns);
+		}
+		mutex_unlock(&ctrl->namespaces_mutex);
+	}
+	if (effects & (NVME_CMD_FX_LBCC | NVME_CMD_FX_CSE_MASK))
+		nvme_unfreeze(ctrl);
+	if (effects & NVME_CMD_FX_CCC)
+		nvme_init_identify(ctrl);
+	if (effects & (NVME_CMD_FX_NIC | NVME_CMD_FX_NCC))
+		nvme_scan(ctrl);
+}
+
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
 	struct nvme_passthru_cmd cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
+	u32 effects = 0;
 	int status;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -997,6 +1057,9 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.flags)
 		return -EINVAL;
 
+	if (ns != NULL)
+		nvme_get_admin_effects(ctrl, cmd.opcode, &effects);
+
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = cmd.opcode;
 	c.common.flags = cmd.flags;
@@ -1013,10 +1076,13 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
+	nvme_passthru_start(ctrl, effects);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			(void __user *)(uintptr_t)cmd.addr, cmd.data_len,
 			(void __user *)(uintptr_t)cmd.metadata, cmd.metadata,
 			0, &cmd.result, timeout);
+	nvme_passthru_end(ctrl, effects);
+
 	if (status >= 0) {
 		if (put_user(cmd.result, &ucmd->result))
 			return -EFAULT;
@@ -1762,6 +1828,37 @@ static void nvme_init_subnqn(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	memset(ctrl->subnqn + off, 0, sizeof(ctrl->subnqn) - off);
 }
 
+static int nvme_get_log(struct nvme_ctrl *ctrl, u8 log_page, void *log, size_t size)
+{
+	struct nvme_command c = { };
+
+
+	c.common.opcode = nvme_admin_get_log_page;
+	c.common.nsid = cpu_to_le32(NVME_NSID_ALL);
+	c.common.cdw10[0] = nvme_get_log_dw10(log_page, size);
+
+	return nvme_submit_sync_cmd(ctrl->admin_q, &c, log, size);
+}
+
+static int nvme_get_effects_log(struct nvme_ctrl *ctrl)
+{
+	int ret;
+
+	if (!ctrl->effects)
+		ctrl->effects = kzalloc(sizeof(*ctrl->effects), GFP_KERNEL);
+
+	if (!ctrl->effects)
+		return 0;
+
+	ret = nvme_get_log(ctrl, NVME_LOG_CMD_FX, ctrl->effects,
+					sizeof(*ctrl->effects));
+	if (ret) {
+		kfree(ctrl->effects);
+		ctrl->effects = NULL;
+	}
+	return ret;
+}
+
 /*
  * Initialize the cached copies of the Identify data and various controller
  * register in our nvme_ctrl structure.  This should be called as soon as
@@ -1797,6 +1894,12 @@ int nvme_init_identify(struct nvme_ctrl *ctrl)
 		return -EIO;
 	}
 
+	if (id->lpa & NVME_CTRL_LPA_CMD_FX_LOG) {
+		ret = nvme_get_effects_log(ctrl);
+		if (ret < 0)
+			return ret;
+	}
+
 	nvme_init_subnqn(ctrl, id);
 
 	if (!ctrl->identified) {
@@ -2522,10 +2625,8 @@ static void nvme_scan_ns_sequential(struct nvme_ctrl *ctrl, unsigned nn)
 	nvme_remove_invalid_namespaces(ctrl, nn);
 }
 
-static void nvme_scan_work(struct work_struct *work)
+static void nvme_scan(struct nvme_ctrl *ctrl)
 {
-	struct nvme_ctrl *ctrl =
-		container_of(work, struct nvme_ctrl, scan_work);
 	struct nvme_id_ctrl *id;
 	unsigned nn;
 
@@ -2549,6 +2650,14 @@ static void nvme_scan_work(struct work_struct *work)
 	kfree(id);
 }
 
+static void nvme_scan_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, scan_work);
+
+	nvme_scan(ctrl);
+}
+
 void nvme_queue_scan(struct nvme_ctrl *ctrl)
 {
 	/*
@@ -2615,18 +2724,13 @@ static bool nvme_ctrl_pp_status(struct nvme_ctrl *ctrl)
 
 static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl)
 {
-	struct nvme_command c = { };
 	struct nvme_fw_slot_info_log *log;
 
 	log = kmalloc(sizeof(*log), GFP_KERNEL);
 	if (!log)
 		return;
 
-	c.common.opcode = nvme_admin_get_log_page;
-	c.common.nsid = cpu_to_le32(NVME_NSID_ALL);
-	c.common.cdw10[0] = nvme_get_log_dw10(NVME_LOG_FW_SLOT, sizeof(*log));
-
-	if (!nvme_submit_sync_cmd(ctrl->admin_q, &c, log, sizeof(*log)))
+	if (nvme_get_log(ctrl, NVME_LOG_FW_SLOT, log, sizeof(*log)))
 		dev_warn(ctrl->device,
 				"Get FW SLOT INFO log error\n");
 	kfree(log);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index d3f3c4447515..69dd8d2cf05c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -172,6 +172,7 @@ struct nvme_ctrl {
 	bool subsystem;
 	unsigned long quirks;
 	struct nvme_id_power_state psd[32];
+	struct nvme_effects_log *effects;
 	struct work_struct scan_work;
 	struct work_struct async_event_work;
 	struct delayed_work ka_work;
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9310ce77d8e1..cad7bde44f8e 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -267,6 +267,7 @@ enum {
 	NVME_CTRL_OACS_SEC_SUPP                 = 1 << 0,
 	NVME_CTRL_OACS_DIRECTIVES		= 1 << 5,
 	NVME_CTRL_OACS_DBBUF_SUPP		= 1 << 8,
+	NVME_CTRL_LPA_CMD_FX_LOG		= 1 << 1,
 };
 
 struct nvme_lbaf {
@@ -396,6 +397,22 @@ struct nvme_fw_slot_info_log {
 };
 
 enum {
+	NVME_CMD_FX_CSUPP	= 1 << 0,
+	NVME_CMD_FX_LBCC	= 1 << 1,
+	NVME_CMD_FX_NCC		= 1 << 2,
+	NVME_CMD_FX_NIC		= 1 << 3,
+	NVME_CMD_FX_CCC		= 1 << 4,
+	NVME_CMD_FX_CSE_MASK	= 3 << 16,
+};
+
+struct nvme_effects_log {
+	__le32 acs[256];
+	__le32 iocs[256];
+	__u8   resv[2048];
+};
+
+
+enum {
 	NVME_SMART_CRIT_SPARE		= 1 << 0,
 	NVME_SMART_CRIT_TEMPERATURE	= 1 << 1,
 	NVME_SMART_CRIT_RELIABILITY	= 1 << 2,
@@ -712,6 +729,7 @@ enum {
 	NVME_LOG_ERROR		= 0x01,
 	NVME_LOG_SMART		= 0x02,
 	NVME_LOG_FW_SLOT	= 0x03,
+	NVME_LOG_CMD_FX		= 0x05,
 	NVME_LOG_DISC		= 0x70,
 	NVME_LOG_RESERVATION	= 0x80,
 	NVME_FWACT_REPL		= (0 << 3),
--


From hch at lst.de  Fri Oct 27 22:52:58 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 07:52:58 +0200
Subject: [PATCH urgent-4.14] nvme: Fix setting logical block format
 when revalidating
In-Reply-To: <20171027195122.1810-1-keith.busch@intel.com>
References: <20171027195122.1810-1-keith.busch@intel.com>
Message-ID: <20171028055258.GA13348@lst.de>

On Fri, Oct 27, 2017 at 01:51:22PM -0600, Keith Busch wrote:
> Revalidating the disk needs to set the logical block format and capacity,
> otherwise it can't figure out if the users modified anything about
> the namespace.
> 
> Fixes: cdbff4f26bd9 ("nvme: remove nvme_revalidate_ns")
> 
> Signed-off-by: Keith Busch <keith.busch at intel.com>

Looks fine to me:

Reviewed-by: Christoph Hellwig <hch at lst.de>

Jens: can you pick this up directly before sending a block PR today?


From hch at lst.de  Fri Oct 27 23:13:42 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 08:13:42 +0200
Subject: [PATCH 04/17] block: add a blk_steal_bios helper
In-Reply-To: <ac15137b-b1a4-3988-17cc-fd6c25749542@mellanox.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-5-hch@lst.de>
 <ac15137b-b1a4-3988-17cc-fd6c25749542@mellanox.com>
Message-ID: <20171028061342.GA13723@lst.de>

On Tue, Oct 24, 2017 at 11:44:26AM +0300, Max Gurtovoy wrote:
>> + * Steal bios from a request.  The request must not have been partially
>> + * completed before.
>> + */
>
> Maybe we can add to the comment that "list" is the destination for the 
> stolen bio.

Sure.

>> +void blk_steal_bios(struct bio_list *list, struct request *rq)
>> +{
>> +	if (rq->bio) {
>> +		if (list->tail)
>> +			list->tail->bi_next = rq->bio;
>> +		else
>> +			list->head = rq->bio;
>> +		list->tail = rq->biotail;
>
> if list->tail != NULL don't we lose the "list->tail->bi_next = rq->bio;" 
> assignment after assigning "list->tail = rq->biotail;" ?

the biolists are a little weird, they are a single linked list of bi_next
plus a tail pointer.

So if the list is emptry (->tail == NULL) we assign the biolist
to ->head and point ->tail to end of the list in the request.

But if the list is not empty we let ->bi_next of the last entry
(as found in ->tail) point to the list we splice on, and still update
->tail to end of the list we spliced on.  So I think this looks all ok.

>> +	}
>> +
>> +	rq->bio = NULL;
>
> we can add this NULL assignment inside the big "if", but I'm not sure 
> regarding the next 2 assignments.
> Anyway not a big deal.

We can move both the ->bio and ->biotail assignments, and I've done it.


From hch at lst.de  Fri Oct 27 23:15:23 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 08:15:23 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <09f8a87b-f8da-9f5a-6efd-f5e9acfaee45@suse.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de>
 <09f8a87b-f8da-9f5a-6efd-f5e9acfaee45@suse.de>
Message-ID: <20171028061523.GB13723@lst.de>

On Tue, Oct 24, 2017 at 09:18:47AM +0200, Hannes Reinecke wrote:
> Can we have some information in sysfs to figure out if a gendisk is
> hidden? I'd hate having to do an inverse lookup in /proc/partitions;
> it's always hard to prove that something is _not_ present.
> And we already present various information (like disk_removable_show()),
> so it's not without precedent.
> And it would make integration with systemd/udev easier.

Sure, I'll add a hidden flag sysfs file.


From hch at lst.de  Fri Oct 27 23:20:41 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 08:20:41 +0200
Subject: [PATCH 17/17] nvme: also expose the namespace identification
 sysfs files for mpath nodes
In-Reply-To: <ab2161cc-aa24-4137-8a68-08e63adce764@suse.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-18-hch@lst.de>
 <ab2161cc-aa24-4137-8a68-08e63adce764@suse.de>
Message-ID: <20171028062041.GC13723@lst.de>

On Tue, Oct 24, 2017 at 09:45:44AM +0200, Hannes Reinecke wrote:
> device_add_disk_with_groups()?
> What happened to that proposal?

Doesn't look like it got anywhere near a tree I could base this work
on.

And what happen to not fullquoting a patch for a two line comment?


From hch at lst.de  Fri Oct 27 23:32:52 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 08:32:52 +0200
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <104278cd-0383-71e0-45c2-750108723b8b@suse.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
 <104278cd-0383-71e0-45c2-750108723b8b@suse.de>
Message-ID: <20171028063252.GA13998@lst.de>

> Sure this works?

It does.

> nvme_req_needs_retry() checks blk_noretry_request():

> The original idea here was to _set_ these bits on multipath path devices
> so that they won't attempt any retry, but rather forward the I/O error
> to the multipath device itself for failover.
> So if these bits are set (as they should be for multipathed devices)
> we'll never attempt any failover...

While that might have been the "original" idea, it isn't what this code
does.  We never set any of the REQ_FAILFAST_* bits in
nvme_ns_head_make_request.  In NVMe there aren't really any device
equivalents of REQ_FAILFAST_ that make sense for multipath.  The only
one that we map to is the limited retry bit, and that is media centric,
so a failver would not help.


From hch at lst.de  Fri Oct 27 23:38:25 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 08:38:25 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171024213959.GB23307@redhat.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
Message-ID: <20171028063825.GA14261@lst.de>

On Tue, Oct 24, 2017 at 05:40:00PM -0400, Mike Snitzer wrote:
> Having the NVme driver go to such lengths to hide its resources from
> upper layers is certainly the work of an evil genius experiencing some
> serious territorial issues.  Not sugar-coating it.. you wouldn't.

I'm pretty surre Hannes will appreciate being called an evil genius :)

> I kept meaning to reply to your earlier iterations on this series to
> ask: can we please get a CONFIG_NVME_MULTIPATHING knob to make it so
> that the NVMe driver doesn't implicitly consume (and hide) all
> per-controler devices?

I thought about adding it, but mostly for a different reason: it's
quite a bit of code, and we now start to see NVMe in deeply embedded
contexts, e.g. the latest Compact Flash spec is based on NVMe, so it
might be a good idea to give people a chance to avoid the overhead.

> Ah well.  There is only one correct way to do NVMe multipathing after
> all right?

I don't think you'll get very useful results, even if you try.  But I
guess we'll just have to tell people to use SuSE if they want NVMe
multipathing to work then :)


From hch at infradead.org  Fri Oct 27 23:45:08 2017
From: hch at infradead.org (Christoph Hellwig)
Date: Fri, 27 Oct 2017 23:45:08 -0700
Subject: [PATCH] nvme: freeze IO accesses around format
In-Reply-To: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
References: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
Message-ID: <20171028064508.GA20977@infradead.org>

And what about sanitize?  I guess instead of looking at opcodes directly
we'd better look at the indications in the Commands Supported and Effects
log.  That's what the Windows driver does, btw.


From guanjunxiong at huawei.com  Sat Oct 28 00:20:07 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Sat, 28 Oct 2017 15:20:07 +0800
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171028063825.GA14261@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de>
Message-ID: <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>

Hi Christoph,

On 2017/10/28 14:38, Christoph Hellwig wrote:
> On Tue, Oct 24, 2017 at 05:40:00PM -0400, Mike Snitzer wrote:
>> Having the NVme driver go to such lengths to hide its resources from
>> upper layers is certainly the work of an evil genius experiencing some
>> serious territorial issues.  Not sugar-coating it.. you wouldn't.
> 
> I'm pretty surre Hannes will appreciate being called an evil genius :)
> 
>> I kept meaning to reply to your earlier iterations on this series to
>> ask: can we please get a CONFIG_NVME_MULTIPATHING knob to make it so
>> that the NVMe driver doesn't implicitly consume (and hide) all
>> per-controler devices?
> 
> I thought about adding it, but mostly for a different reason: it's
> quite a bit of code, and we now start to see NVMe in deeply embedded
> contexts, e.g. the latest Compact Flash spec is based on NVMe, so it
> might be a good idea to give people a chance to avoid the overhead.
> 

Think of some current advanced features of DM-Multipath combined with multipath-tools
such as path-latency priority grouping, intermittent IO error accounting for path
degradation, delayed or immediate or follow-over failback feature.
Those features, which is significant in some scenario, need to use per-controller block devices.
Therefore, I think it is worthy adding a CONFIG_NVME_MULTIPATHING knob to
hide or show per-controller block devices.

How about let me to add this this CONFIG_NVME_MULTIPATHING knob?

Regards
Guan



>> Ah well.  There is only one correct way to do NVMe multipathing after
>> all right?
> 
> I don't think you'll get very useful results, even if you try.  But I
> guess we'll just have to tell people to use SuSE if they want NVMe
> multipathing to work then :)
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
> 
> 



From hch at lst.de  Sat Oct 28 00:42:45 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sat, 28 Oct 2017 09:42:45 +0200
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de>
 <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>
Message-ID: <20171028074245.GA15510@lst.de>

On Sat, Oct 28, 2017 at 03:20:07PM +0800, Guan Junxiong wrote:
> Think of some current advanced features of DM-Multipath combined with multipath-tools
> such as path-latency priority grouping, intermittent IO error accounting for path
> degradation, delayed or immediate or follow-over failback feature.
> Those features, which is significant in some scenario, need to use per-controller block devices.
> Therefore, I think it is worthy adding a CONFIG_NVME_MULTIPATHING knob to
> hide or show per-controller block devices.
> 
> How about let me to add this this CONFIG_NVME_MULTIPATHING knob?

There is defintively not going to be a run-time nob, sorry.  You've
spent this work on dm-multipath after the nvme group pretty clearly
said that this is not what we are going to support, and we have not
interest in supporting this.  Especially as proper path discovery,
asymetic namespaces states and similar can only be supported properly
with the in-kernel code.


From guanjunxiong at huawei.com  Sat Oct 28 03:09:46 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Sat, 28 Oct 2017 18:09:46 +0800
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171028074245.GA15510@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de>
 <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>
 <20171028074245.GA15510@lst.de>
Message-ID: <99f3c781-71ce-f915-d6f4-88d6dd115876@huawei.com>



On 2017/10/28 15:42, Christoph Hellwig wrote:
> On Sat, Oct 28, 2017 at 03:20:07PM +0800, Guan Junxiong wrote:
>> Think of some current advanced features of DM-Multipath combined with multipath-tools
>> such as path-latency priority grouping, intermittent IO error accounting for path
>> degradation, delayed or immediate or follow-over failback feature.
>> Those features, which is significant in some scenario, need to use per-controller block devices.
>> Therefore, I think it is worthy adding a CONFIG_NVME_MULTIPATHING knob to
>> hide or show per-controller block devices.
>>
>> How about let me to add this this CONFIG_NVME_MULTIPATHING knob?
> 
> There is defintively not going to be a run-time nob, sorry.  You've
> spent this work on dm-multipath after the nvme group pretty clearly
> said that this is not what we are going to support, and we have not
> interest in supporting this.  Especially as proper path discovery,
> asymetic namespaces states and similar can only be supported properly
> with the in-kernel code.
> 
Does it mean some extra works such as:
1) showing the path topology of nvme multipath device
2) daemon to implement immediate and delayed failback
3) detecting sub-healthy path due to shaky link
4) grouping paths besides ANA
...
and so on,
need to integrate into the user-space tool such as nvme-cli
or a new invented user-space tool named "nvme-mpath" ?
Which do you prefer?

And the kernel also needs to export more setting and getting methods.

Regards
Guan
.




From hare at suse.de  Sat Oct 28 06:34:39 2017
From: hare at suse.de (Hannes Reinecke)
Date: Sat, 28 Oct 2017 15:34:39 +0200
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <352f980c-2422-f917-a551-99bedf27e0ab@broadcom.com>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
 <e542bb4d-946f-a080-5701-f8019fdbc752@suse.de>
 <352f980c-2422-f917-a551-99bedf27e0ab@broadcom.com>
Message-ID: <f0300e47-e219-19f9-1495-a662f9062b4d@suse.de>

On 10/27/2017 07:50 PM, James Smart wrote:
> On 10/27/2017 12:48 AM, Hannes Reinecke wrote:
>> On 10/09/2017 10:10 PM, James Smart wrote:
>>> Sagi recently posted a "one-shot" call to nvme-connect via a systemd
>>> script. FC has no problems operating with it assuming all the required
>>> connect fields are present (host_traddr required).
>>>
>>> However FC is a bit more dynamic, and we've built up scripts that can
>>> be triggered whenever a new FC remote port with a discovery subsytem is
>>> detected. Whenever such a port is detected (by registration of the
>>> remoteport with the nvme_fc transport by the lldd), the transport
>>> generates a udev event.? A udev rule is created that parses the event
>>> information into the FC remote port (traddr) that supports a discovery
>>> subsystem and the FC host port that is connected to the FC remote port
>>> (host_traddr). The rule then invokes an instance-based systemd service
>>> to perform a "nvme connect-all" with the traddr and host-traddr info.
>>>
>>> To resolve some issues seen by discovery of remote ports before the udev
>>> service was running and where the events were not replayed, there is an
>>> additional systemd service that runs once and invokes a script which
>>> locates the FC host ports on the system, looks for any connected FC
>>> remote ports with discovery subsystems, and invokes a connect-all for
>>> them.? The FC port information is specific to lpfc attributes currently,
>>> but when the shared transport is in place, they will be common.
>>>
>>> I don't claim to be a systemd expert, so comments are certainly welcome.
>>>
>> In general I'm not in favour of calling 'systemctl start' from a udev
>> rule; this typically will cause systemd to rearrange the internal
>> dependency chain, and might trigger various services to be enabled or
>> disabled.
>>
>> The 'normal' flow of events is:
>>
>> kernel device -> uevent -> udev rules -> systemd services
>>
>> ie for the autoconnect it would be _far_ better if we could have an
>> event per NVMe lport device, as then we could call 'connect' on the
>> lport device.
>>
>> So a better workflow would be to have the 'discovery' event trigger a
>> 'change' (or 'add'?) event on the lports, and then we could have
>> per-lport udev rules triggering the actualy connect.
>>
>> Let's see if I can come up with something along these lines.
>>
>> Cheers,
>>
>> Hannes
> 
> I'm certainly open to better options. udev/systemd is far from nice.?
> However, what I did has quite a bit of precedence.
> 
> I have no idea what you would scan on by just an lport. It's not enough
> for nvme-cli. You must have the lport and the rport information, which
> is passed via the udev event info and which must also be passed to nvme
> connect-all
> 
Oh, I know that.
What I would prefer is to have a single udev rule per nvme connection,
containing both lport and rport information.
That would relieve us of having additional configuration, and we can
enable/disable connection by just having that udev rule.
This works pretty well; we're doing it exactly the same way on S/390 for
the network and zfcp connections.
So if we have that then we only need to know when the lport/rport
combination is ready; once that is done we can call the udev rule.
Ideally we'll be getting an event containing both of these informations,
then udev will be selecting the correct rule automatically.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From snitzer at redhat.com  Sat Oct 28 07:17:45 2017
From: snitzer at redhat.com (Mike Snitzer)
Date: Sat, 28 Oct 2017 10:17:45 -0400
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171028063825.GA14261@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de>
 <20171024213959.GB23307@redhat.com> <20171028063825.GA14261@lst.de>
Message-ID: <20171028141745.GA32039@redhat.com>

On Sat, Oct 28 2017 at  2:38am -0400,
Christoph Hellwig <hch at lst.de> wrote:

> On Tue, Oct 24, 2017 at 05:40:00PM -0400, Mike Snitzer wrote:
> > Having the NVme driver go to such lengths to hide its resources from
> > upper layers is certainly the work of an evil genius experiencing some
> > serious territorial issues.  Not sugar-coating it.. you wouldn't.
> 
> I'm pretty surre Hannes will appreciate being called an evil genius :)

Well, to be fair.. it doesn't take that much brain power to arrive at
isolationism.

And pretty sure Hannes had a better idea with using the blkdev_get
(holders/claim) interface but you quickly dismissed that.  Despite it
being the best _existing_ way to ensure mutual exclusion with the rest
of the block layer.

> > I kept meaning to reply to your earlier iterations on this series to
> > ask: can we please get a CONFIG_NVME_MULTIPATHING knob to make it so
> > that the NVMe driver doesn't implicitly consume (and hide) all
> > per-controler devices?
> 
> I thought about adding it, but mostly for a different reason: it's
> quite a bit of code, and we now start to see NVMe in deeply embedded
> contexts, e.g. the latest Compact Flash spec is based on NVMe, so it
> might be a good idea to give people a chance to avoid the overhead.

OK, so please add it.

> > Ah well.  There is only one correct way to do NVMe multipathing after
> > all right?
> 
> I don't think you'll get very useful results, even if you try.  But I
> guess we'll just have to tell people to use SuSE if they want NVMe
> multipathing to work then :)

Don't do that.  Don't assume you know it all.  Don't fabricate vendor
wars in your head and then project it out to the rest of the community.
We're all in this together.

Fact is Hannes and I have exchanged private mail (in response to this
very thread) and we agree that your approach is currently not suitable
for enterprise deployment.  Hannes needs to also deal with the coming
duality of Linux multipathing and you aren't making it easy.  Just
because you're able to be myopic doesn't mean the rest of us with way
more direct customers behind us can be.

You're finding your way with this new multipath model and I'm happy to
see that happen.  But what I'm not happy about is the steps you're
taking to be actively disruptive.  There were so many ways this all
could've gone and sadly you've elected to stand up an architecture that
doesn't even allow the prospect of reuse.  And for what?  Because doing
so takes 10% more work?  Well we can backfill that work if you'll grant
us an open-mind.

I'm really not against you.  I just need very basic controls put into
the NVMe multipathing code that allows it to be disabled (yet reused).
Not that I have immediate plans to actually _use_ it.  My hope is I can
delegate NVMe multipathing to NVMe!  But I need the latitude to find my
way with the requirements I am, or will be, needing to consider.

Please don't paint me and others in my position into a corner.

Mike


From james.smart at broadcom.com  Sat Oct 28 09:08:50 2017
From: james.smart at broadcom.com (James Smart)
Date: Sat, 28 Oct 2017 09:08:50 -0700
Subject: [PATCH] nvme_fc auto-connect scripts
In-Reply-To: <f0300e47-e219-19f9-1495-a662f9062b4d@suse.de>
References: <20171009201032.2018-1-jsmart2021@gmail.com>
 <e542bb4d-946f-a080-5701-f8019fdbc752@suse.de>
 <352f980c-2422-f917-a551-99bedf27e0ab@broadcom.com>
 <f0300e47-e219-19f9-1495-a662f9062b4d@suse.de>
Message-ID: <159cfcc6-40e8-9062-7341-9a3de5f177af@broadcom.com>

On 10/28/2017 6:34 AM, Hannes Reinecke wrote:
>
> Oh, I know that.
> What I would prefer is to have a single udev rule per nvme connection,
> containing both lport and rport information.
> That would relieve us of having additional configuration, and we can
> enable/disable connection by just having that udev rule.
> This works pretty well; we're doing it exactly the same way on S/390 for
> the network and zfcp connections.
> So if we have that then we only need to know when the lport/rport
> combination is ready; once that is done we can call the udev rule.
> Ideally we'll be getting an event containing both of these informations,
> then udev will be selecting the correct rule automatically.
>
> Cheers,
>
> Hannes

The main reason for the systemd use was the potential duration of the 
resulting nvme connect-all. The scripts are simple if udev makes the 
nvme cli call and don't require anything other than a single generic 
rule. But given its potential stalls, very bad.

If I understand you correctly, what you describe requires apriori 
knowledge of the connections in order to add the udevs rule. This is a 
step backward from current FC behavior and what users are used to.

-- james



From jsmart2021 at gmail.com  Sat Oct 28 10:21:09 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:09 -0700
Subject: [PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan
Message-ID: <20171028172114.2045-1-jsmart2021@gmail.com>

A transport may have a transport-specific mechanism that can signal
when discovery controller content has changed and request a host
to reconnect to the discovery controller.

FC is such a transport. RSCNs may be generated by the FC port with
the discovery server, with the RSCNs then broadcast to the FC-NVME
hosts. A host, upon receiving the RSCN, would validate connectivity
then initiate a discovery controller rescan, allowing new subsystems
to be connected to or updating subsystem connectivity tables.

These patches:
- Modify the nvmet core layer to call a transport callback on every
  subsystem add or remove from a transport port.
- Modify the nvmet-fc transport to support the callback, and add its
  own internal lldd api to generate RSCN's via the lldd.
- Modify the lpfc driver to send/receive RSCNs for FC-NVME: transmit
  the changed attribute RSCN on the target, receiving the RSCN on
  the initiator and invoking the nvmet-fc transport rescan api.
  Also adds manual sysfs mechanism to generate the RSCN on the target.


Dick Kennedy (1):
  lpfc: Add sysfs interface to post NVME RSCN

James Smart (4):
  nvmet: call transport on subsystem add and delete
  nvmet_fc: support transport subsystem events
  lpfc: Add support to generate RSCN events for nport
  lpfc: Add NVME rescan support via RSCNs

 drivers/nvme/target/configfs.c   |   2 +
 drivers/nvme/target/core.c       |  10 ++++
 drivers/nvme/target/fc.c         |  10 ++++
 drivers/nvme/target/nvmet.h      |   2 +
 drivers/scsi/lpfc/lpfc.h         |   2 +
 drivers/scsi/lpfc/lpfc_attr.c    |  62 ++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_crtn.h    |   4 ++
 drivers/scsi/lpfc/lpfc_els.c     | 118 +++++++++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_hbadisc.c |  35 ++++++++++++
 drivers/scsi/lpfc/lpfc_hw.h      |   9 +++
 drivers/scsi/lpfc/lpfc_nvme.c    |  42 ++++++++++++++
 drivers/scsi/lpfc/lpfc_nvmet.c   |  18 ++++++
 drivers/scsi/lpfc/lpfc_sli.c     |   1 +
 include/linux/nvme-fc-driver.h   |   6 ++
 14 files changed, 321 insertions(+)

-- 
2.13.1



From jsmart2021 at gmail.com  Sat Oct 28 10:21:10 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:10 -0700
Subject: [PATCH 1/5] nvmet: call transport on subsystem add and delete
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <20171028172114.2045-2-jsmart2021@gmail.com>

A transport may have a transport-specific mechanism that can signal
when discovery controller content has changed and request a host
to reconnect to the discovery controller.

FC is such a transport. RSCNs may be generated by the FC port with
the discovery server, with the RSCNs then broadcast to the FC-NVME
hosts. A host, upon receiving the RSCN, would validate connectivity
then initiate a discovery controller rescan, allowing new subsystems
to be connected to or updating subsystem connectivity tables.

Modify the subsystem link/unlink routines to always call a new
nvmet_port_subsystem_chg() routine whenever a new subsystem is
linked to a port or removed from a port.

The nvmet_port_subsystem_chg() routine will look to see if the
transport supports a discovery change callback, and if so, calls
the callback.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/target/configfs.c |  2 ++
 drivers/nvme/target/core.c     | 10 ++++++++++
 drivers/nvme/target/nvmet.h    |  2 ++
 3 files changed, 14 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index b6aeb1d70951..38adcd7eafb1 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -503,6 +503,7 @@ static int nvmet_port_subsys_allow_link(struct config_item *parent,
 	list_add_tail(&link->entry, &port->subsystems);
 	nvmet_genctr++;
 	up_write(&nvmet_config_sem);
+	nvmet_port_subsystem_chg(port);
 	return 0;
 
 out_free_link:
@@ -529,6 +530,7 @@ static void nvmet_port_subsys_drop_link(struct config_item *parent,
 found:
 	list_del(&p->entry);
 	nvmet_genctr++;
+	nvmet_port_subsystem_chg(port);
 	if (list_empty(&port->subsystems))
 		nvmet_disable_port(port);
 	up_write(&nvmet_config_sem);
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index da088293eafc..f70d900774d0 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -206,6 +206,16 @@ void nvmet_disable_port(struct nvmet_port *port)
 	module_put(ops->owner);
 }
 
+void nvmet_port_subsystem_chg(struct nvmet_port *port)
+{
+	struct nvmet_fabrics_ops *ops;
+
+	ops = nvmet_transports[port->disc_addr.trtype];
+
+	if (ops->discov_chg)
+		ops->discov_chg(port);
+}
+
 static void nvmet_keep_alive_timer(struct work_struct *work)
 {
 	struct nvmet_ctrl *ctrl = container_of(to_delayed_work(work),
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index e342f02845c1..87c7f6e84694 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -209,6 +209,7 @@ struct nvmet_fabrics_ops {
 	int (*add_port)(struct nvmet_port *port);
 	void (*remove_port)(struct nvmet_port *port);
 	void (*delete_ctrl)(struct nvmet_ctrl *ctrl);
+	void (*discov_chg)(struct nvmet_port *port);
 };
 
 #define NVMET_MAX_INLINE_BIOVEC	8
@@ -302,6 +303,7 @@ void nvmet_unregister_transport(struct nvmet_fabrics_ops *ops);
 
 int nvmet_enable_port(struct nvmet_port *port);
 void nvmet_disable_port(struct nvmet_port *port);
+void nvmet_port_subsystem_chg(struct nvmet_port *port);
 
 void nvmet_referral_enable(struct nvmet_port *parent, struct nvmet_port *port);
 void nvmet_referral_disable(struct nvmet_port *port);
-- 
2.13.1



From jsmart2021 at gmail.com  Sat Oct 28 10:21:12 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:12 -0700
Subject: [PATCH 3/5] lpfc: Add support to generate RSCN events for nport
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <20171028172114.2045-4-jsmart2021@gmail.com>

This patch adds general Changed Attribute RSCN support, both transmit
and receive, to the driver. This support will be used with NVME to
request initiators to perform nvme discovery.

This patch:
adds RSCN definitions
adds statistics on RSCN transmissions
creates a routine that will transmit a Changed Attribute RSCN
Handle RSCN receipt

Signed-off-by: Dick Kennedy <dick.kennedy at broadcom.com>
Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/scsi/lpfc/lpfc.h         |   1 +
 drivers/scsi/lpfc/lpfc_crtn.h    |   2 +
 drivers/scsi/lpfc/lpfc_els.c     | 114 +++++++++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_hbadisc.c |  35 ++++++++++++
 drivers/scsi/lpfc/lpfc_hw.h      |   9 ++++
 drivers/scsi/lpfc/lpfc_sli.c     |   1 +
 6 files changed, 162 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc.h b/drivers/scsi/lpfc/lpfc.h
index 8eb3f96fe068..157788f0bc10 100644
--- a/drivers/scsi/lpfc/lpfc.h
+++ b/drivers/scsi/lpfc/lpfc.h
@@ -277,6 +277,7 @@ struct lpfc_stats {
 	uint32_t elsXmitADISC;
 	uint32_t elsXmitLOGO;
 	uint32_t elsXmitSCR;
+	uint32_t elsXmitRSCN;
 	uint32_t elsXmitRNID;
 	uint32_t elsXmitFARP;
 	uint32_t elsXmitFARPR;
diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h
index 7e300734b345..6657b71d5a57 100644
--- a/drivers/scsi/lpfc/lpfc_crtn.h
+++ b/drivers/scsi/lpfc/lpfc_crtn.h
@@ -142,6 +142,7 @@ int lpfc_issue_els_adisc(struct lpfc_vport *, struct lpfc_nodelist *, uint8_t);
 int lpfc_issue_els_logo(struct lpfc_vport *, struct lpfc_nodelist *, uint8_t);
 int lpfc_issue_els_npiv_logo(struct lpfc_vport *, struct lpfc_nodelist *);
 int lpfc_issue_els_scr(struct lpfc_vport *, uint32_t, uint8_t);
+int lpfc_issue_els_rscn(struct lpfc_vport *vport, uint8_t retry);
 int lpfc_issue_fabric_reglogin(struct lpfc_vport *);
 int lpfc_els_free_iocb(struct lpfc_hba *, struct lpfc_iocbq *);
 int lpfc_ct_free_iocb(struct lpfc_hba *, struct lpfc_iocbq *);
@@ -357,6 +358,7 @@ void lpfc_mbox_timeout_handler(struct lpfc_hba *);
 struct lpfc_nodelist *lpfc_findnode_did(struct lpfc_vport *, uint32_t);
 struct lpfc_nodelist *lpfc_findnode_wwpn(struct lpfc_vport *,
 					 struct lpfc_name *);
+struct lpfc_nodelist *lpfc_findnode_mapped(struct lpfc_vport *vport);
 
 int lpfc_sli_issue_mbox_wait(struct lpfc_hba *, LPFC_MBOXQ_t *, uint32_t);
 
diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
index 468a66371de9..936fc22dfcae 100644
--- a/drivers/scsi/lpfc/lpfc_els.c
+++ b/drivers/scsi/lpfc/lpfc_els.c
@@ -2868,6 +2868,109 @@ lpfc_cmpl_els_cmd(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
 }
 
 /**
+ * lpfc_issue_els_rscn - Issue a rscn to an node on a vport
+ * @vport: pointer to a host virtual N_Port data structure.
+ * @retry: number of retries to the command IOCB.
+ *
+ * This routine issues a RSCN to a fabric node on a @vport.
+ *  The remote node @nportid is passed into the function. It
+ * or Pt2Pt topologies, the first MAPPED node will get an RSCN
+ * notification.  For Fabric topologies, the well known SCR_DID
+ * gets notified for broadcast to all zone members.
+ *
+ * Note that, in lpfc_prep_els_iocb() routine, the reference count of ndlp
+ * will be incremented by 1 for holding the ndlp and the reference to ndlp
+ * will be stored into the context1 field of the IOCB for the completion
+ * callback function to the RSCN ELS command.
+ *
+ * Return code
+ *   0 - Successfully issued RSCN command
+ *   1 - Failed to issue RSCN command
+ **/
+int
+lpfc_issue_els_rscn(struct lpfc_vport *vport, uint8_t retry)
+{
+	struct lpfc_hba  *phba = vport->phba;
+	struct lpfc_iocbq *elsiocb;
+	uint8_t *pcmd;
+	uint16_t cmdsize;
+	uint32_t value, nportid;
+	struct lpfc_nodelist *ndlp;
+
+	/* Not supported for private loop */
+	if (phba->fc_topology == LPFC_TOPOLOGY_LOOP &&
+	    !(vport->fc_flag & FC_PUBLIC_LOOP))
+		return 1;
+
+	cmdsize = (sizeof(uint32_t) + sizeof(SCR));
+	if (vport->fc_flag & FC_PT2PT) {
+		ndlp = lpfc_findnode_mapped(vport);
+		if (!ndlp)
+			return 1;
+	} else {
+		nportid = SCR_DID;
+
+		ndlp = lpfc_findnode_did(vport, nportid);
+		if (!ndlp) {
+			ndlp = lpfc_nlp_init(vport, nportid);
+			if (!ndlp)
+				return 1;
+			lpfc_enqueue_node(vport, ndlp);
+		} else if (!NLP_CHK_NODE_ACT(ndlp)) {
+			ndlp = lpfc_enable_node(vport, ndlp,
+						NLP_STE_UNUSED_NODE);
+			if (!ndlp)
+				return 1;
+		}
+	}
+	elsiocb = lpfc_prep_els_iocb(vport, 1, cmdsize, retry, ndlp,
+				     ndlp->nlp_DID, ELS_CMD_RSCN_XMT);
+
+	if (!elsiocb) {
+		/* This will trigger the release of the node just
+		 * allocated
+		 */
+		lpfc_nlp_put(ndlp);
+		return 1;
+	}
+
+	pcmd = (uint8_t *)(((struct lpfc_dmabuf *)elsiocb->context2)->virt);
+
+	*((uint32_t *)(pcmd)) = ELS_CMD_RSCN_XMT;
+	pcmd += sizeof(uint32_t);
+
+	/* For SCR, remainder of payload is SCR parameter page */
+	memset(pcmd, 0, sizeof(struct fc_RSCN));
+	/* Generic Attribute of 0 is used */
+	value = vport->fc_myDID;
+
+	((struct fc_RSCN *)pcmd)->NPort_ID = be32_to_cpu(value);
+
+	lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_CMD,
+			      "Issue RSCN:       did:x%x",
+			      ndlp->nlp_DID, 0, 0);
+
+	phba->fc_stat.elsXmitRSCN++;
+	elsiocb->iocb_cmpl = lpfc_cmpl_els_cmd;
+	if (lpfc_sli_issue_iocb(phba, LPFC_ELS_RING, elsiocb, 0) ==
+	    IOCB_ERROR) {
+		/* The additional lpfc_nlp_put will cause the following
+		 * lpfc_els_free_iocb routine to trigger the rlease of
+		 * the node.
+		 */
+		lpfc_nlp_put(ndlp);
+		lpfc_els_free_iocb(phba, elsiocb);
+		return 1;
+	}
+	/* This will cause the callback-function lpfc_cmpl_els_cmd to
+	 * trigger the release of node.
+	 */
+
+	lpfc_nlp_put(ndlp);
+	return 0;
+}
+
+/**
  * lpfc_issue_els_scr - Issue a scr to an node on a vport
  * @vport: pointer to a host virtual N_Port data structure.
  * @nportid: N_Port identifier to the remote node.
@@ -6071,6 +6174,16 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb,
 		fc_host_post_event(shost, fc_get_event_number(),
 			FCH_EVT_RSCN, lp[i]);
 
+	/* Check if RSCN is coming from a direct-connected remote NPort */
+	if (vport->fc_flag & FC_PT2PT) {
+		/* If so, just ACC it, no other action needed for now */
+		lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS,
+				 "2024 pt2pt RSCN %08x Data: x%x x%x\n",
+				 *lp, vport->fc_flag, payload_len);
+		lpfc_els_rsp_acc(vport, ELS_CMD_ACC, cmdiocb, ndlp, NULL);
+		return 0;
+	}
+
 	/* If we are about to begin discovery, just ACC the RSCN.
 	 * Discovery processing will satisfy it.
 	 */
@@ -7886,6 +7999,7 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring,
 	elsiocb->context1 = lpfc_nlp_get(ndlp);
 	elsiocb->vport = vport;
 
+	/* Since RSCN can be varying lengths, lets mask that off */
 	if ((cmd & ELS_CMD_MASK) == ELS_CMD_RSCN) {
 		cmd &= ELS_CMD_MASK;
 	}
diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 20808349a80e..3eb37d5070c3 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -5119,6 +5119,41 @@ __lpfc_findnode_did(struct lpfc_vport *vport, uint32_t did)
 }
 
 struct lpfc_nodelist *
+lpfc_findnode_mapped(struct lpfc_vport *vport)
+{
+	struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
+	struct lpfc_nodelist *ndlp;
+	uint32_t data1;
+	unsigned long iflags;
+
+	spin_lock_irqsave(shost->host_lock, iflags);
+
+	list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) {
+		if (ndlp->nlp_state == NLP_STE_UNMAPPED_NODE ||
+		    ndlp->nlp_state == NLP_STE_MAPPED_NODE) {
+			data1 = (((uint32_t)ndlp->nlp_state << 24) |
+				 ((uint32_t)ndlp->nlp_xri << 16) |
+				 ((uint32_t)ndlp->nlp_type << 8) |
+				 ((uint32_t)ndlp->nlp_rpi & 0xff));
+			spin_unlock_irqrestore(shost->host_lock, iflags);
+			lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE,
+					 "2025 FIND node DID "
+					 "Data: x%p x%x x%x x%x %p\n",
+					 ndlp, ndlp->nlp_DID,
+					 ndlp->nlp_flag, data1,
+					 ndlp->active_rrqs_xri_bitmap);
+			return ndlp;
+		}
+	}
+	spin_unlock_irqrestore(shost->host_lock, iflags);
+
+	/* FIND node did <did> NOT FOUND */
+	lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE,
+			 "2026 FIND mapped did NOT FOUND.\n");
+	return NULL;
+}
+
+struct lpfc_nodelist *
 lpfc_findnode_did(struct lpfc_vport *vport, uint32_t did)
 {
 	struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
diff --git a/drivers/scsi/lpfc/lpfc_hw.h b/drivers/scsi/lpfc/lpfc_hw.h
index bdc1f184f67a..a1443592fa46 100644
--- a/drivers/scsi/lpfc/lpfc_hw.h
+++ b/drivers/scsi/lpfc/lpfc_hw.h
@@ -561,6 +561,8 @@ struct fc_vft_header {
 
 /*
  *  Extended Link Service LS_COMMAND codes (Payload Word 0)
+ *  The supported payload size is also hard-coded into this word
+ *  when necessary.
  */
 #ifdef __BIG_ENDIAN_BITFIELD
 #define ELS_CMD_MASK      0xffff0000
@@ -597,6 +599,7 @@ struct fc_vft_header {
 #define ELS_CMD_RPS       0x56000000
 #define ELS_CMD_RPL       0x57000000
 #define ELS_CMD_FAN       0x60000000
+#define ELS_CMD_RSCN_XMT  0x61040008
 #define ELS_CMD_RSCN      0x61040000
 #define ELS_CMD_SCR       0x62000000
 #define ELS_CMD_RNID      0x78000000
@@ -637,6 +640,7 @@ struct fc_vft_header {
 #define ELS_CMD_RPS       0x56
 #define ELS_CMD_RPL       0x57
 #define ELS_CMD_FAN       0x60
+#define ELS_CMD_RSCN_XMT  0x08000461
 #define ELS_CMD_RSCN      0x0461
 #define ELS_CMD_SCR       0x62
 #define ELS_CMD_RNID      0x78
@@ -1088,6 +1092,11 @@ struct fc_lcb_res_frame {
 	uint16_t      lcb_duration;     /* LCB Payload Word 2, bit 15:0  */
 };
 
+struct fc_RSCN {	/* RSCN ELS frame */
+#define RSCN_EQ_PORT_ATTRIBUTE	2
+	uint32_t NPort_ID;
+};
+
 /*
  * Read Diagnostic Parameters (RDP) ELS frame.
  */
diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c
index 8b119f87b51d..53a4ac67c9e1 100644
--- a/drivers/scsi/lpfc/lpfc_sli.c
+++ b/drivers/scsi/lpfc/lpfc_sli.c
@@ -8761,6 +8761,7 @@ lpfc_sli4_iocb2wqe(struct lpfc_hba *phba, struct lpfc_iocbq *iocbq,
 		if (if_type == LPFC_SLI_INTF_IF_TYPE_2) {
 			if (pcmd && (*pcmd == ELS_CMD_FLOGI ||
 				*pcmd == ELS_CMD_SCR ||
+				*pcmd == ELS_CMD_RSCN_XMT ||
 				*pcmd == ELS_CMD_FDISC ||
 				*pcmd == ELS_CMD_LOGO ||
 				*pcmd == ELS_CMD_PLOGI)) {
-- 
2.13.1



From jsmart2021 at gmail.com  Sat Oct 28 10:21:13 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:13 -0700
Subject: [PATCH 4/5] lpfc: Add NVME rescan support via RSCNs
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <20171028172114.2045-5-jsmart2021@gmail.com>

This patch adds NVME rescan support via RSCNs

NVME Target:
  Adds support for the new nvme_subsystem_change callback. The
  callback can be invoked by the nvmet_fc transport when it detects
  conditions that would like to have nvme discovery invoked again.
  Typically this is upon the addition of new subsystems added to
  the nvmet configuration (thus the transport tying the callback
  being invoked to port add calls).
  The callback routine will generate an RSCN to the fabric in
  order to cause RSCN events to all initiators with view of the
  nport.

NVME Host:
  Upon reception of an RSCN event for an nport which supports
  NVME initiator and which is currently logged in and PRLI'd,
  the initiator will call the nvme_fc transport
  nvme_fc_rescan_remoteport() routine to request nvme discovery
  to be performed again on the port pair.

Signed-off-by: Dick Kennedy <dick.kennedy at broadcom.com>
Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/scsi/lpfc/lpfc_crtn.h  |  2 ++
 drivers/scsi/lpfc/lpfc_els.c   |  4 ++++
 drivers/scsi/lpfc/lpfc_nvme.c  | 42 ++++++++++++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_nvmet.c | 18 ++++++++++++++++++
 4 files changed, 66 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h
index 6657b71d5a57..86f034fefa46 100644
--- a/drivers/scsi/lpfc/lpfc_crtn.h
+++ b/drivers/scsi/lpfc/lpfc_crtn.h
@@ -546,6 +546,8 @@ int lpfc_sli4_dump_page_a0(struct lpfc_hba *phba, struct lpfcMboxq *mbox);
 void lpfc_mbx_cmpl_rdp_page_a0(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb);
 
 /* NVME interfaces. */
+void lpfc_nvme_rescan_port(struct lpfc_vport *vport,
+			   struct lpfc_nodelist *ndlp);
 void lpfc_nvme_unregister_port(struct lpfc_vport *vport,
 			struct lpfc_nodelist *ndlp);
 int lpfc_nvme_register_port(struct lpfc_vport *vport,
diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
index 936fc22dfcae..b02684113c75 100644
--- a/drivers/scsi/lpfc/lpfc_els.c
+++ b/drivers/scsi/lpfc/lpfc_els.c
@@ -6071,6 +6071,10 @@ lpfc_rscn_recovery_check(struct lpfc_vport *vport)
 		if (vport->phba->nvmet_support)
 			continue;
 
+		/* Check to see if we need to NVME rescan this remoteport. */
+		if (ndlp->nlp_fc4_type & NLP_FC4_NVME)
+			lpfc_nvme_rescan_port(vport, ndlp);
+
 		lpfc_disc_state_machine(vport, ndlp, NULL,
 					NLP_EVT_DEVICE_RECOVERY);
 		lpfc_cancel_retry_delay_tmo(vport, ndlp);
diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c
index 23bdb1ca106e..b372da3672eb 100644
--- a/drivers/scsi/lpfc/lpfc_nvme.c
+++ b/drivers/scsi/lpfc/lpfc_nvme.c
@@ -2375,6 +2375,48 @@ lpfc_nvme_register_port(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp)
 #endif
 }
 
+/* lpfc_nvme_rescan_port - Check to see if we should rescan this remoteport
+ *
+ * If the ndlp represents an NVME Target, that we are logged into,
+ * ping the NVME FC Transport layer to initiate a device rescan
+ * on this remote NPort.
+ */
+void
+lpfc_nvme_rescan_port(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp)
+{
+#if (IS_ENABLED(CONFIG_NVME_FC))
+	struct lpfc_nvme_rport *rport;
+	struct nvme_fc_remote_port *remoteport;
+
+	rport = ndlp->nrport;
+
+	lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_DISC,
+			 "6170 Rescan NPort DID x%06x type x%x state x%x rport %p\n",
+			 ndlp->nlp_DID, ndlp->nlp_type, ndlp->nlp_state, rport);
+	if (!rport)
+		goto input_err;
+	remoteport = rport->remoteport;
+	if (!remoteport)
+		goto input_err;
+
+	/* Only rescan if we are an NVME target in the MAPPED state */
+	if (remoteport->port_role & FC_PORT_ROLE_NVME_TARGET &&
+	    ndlp->nlp_state == NLP_STE_MAPPED_NODE) {
+		nvme_fc_rescan_remoteport(remoteport);
+
+		lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC,
+				 "6172 NVME rescanned DID x%06x "
+				 "port_state x%x\n",
+				 ndlp->nlp_DID, remoteport->port_state);
+	}
+input_err:
+	return;
+#endif
+	lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC,
+			 "6169 State error: lport %p, rport%p FCID x%06x\n",
+			 vport->localport, ndlp->rport, ndlp->nlp_DID);
+}
+
 /* lpfc_nvme_unregister_port - unbind the DID and port_role from this rport.
  *
  * There is no notion of Devloss or rport recovery from the current
diff --git a/drivers/scsi/lpfc/lpfc_nvmet.c b/drivers/scsi/lpfc/lpfc_nvmet.c
index 0b7c1a49e203..51888192eed7 100644
--- a/drivers/scsi/lpfc/lpfc_nvmet.c
+++ b/drivers/scsi/lpfc/lpfc_nvmet.c
@@ -866,6 +866,23 @@ lpfc_nvmet_defer_rcv(struct nvmet_fc_target_port *tgtport,
 	lpfc_rq_buf_free(phba, &nvmebuf->hbuf); /* repost */
 }
 
+static void
+lpfc_nvmet_subsystem_change(struct nvmet_fc_target_port *tgtport)
+{
+	struct lpfc_nvmet_tgtport *tgtp;
+	struct lpfc_hba *phba;
+	uint32_t rc;
+
+	tgtp = tgtport->private;
+	phba = tgtp->phba;
+
+	rc = lpfc_issue_els_rscn(phba->pport, 0);
+	lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
+			"6420 NVMET subsystem change: "
+			"Notification %s\n",
+			(rc) ? "Failed" : "Sent");
+}
+
 static struct nvmet_fc_target_template lpfc_tgttemplate = {
 	.targetport_delete = lpfc_nvmet_targetport_delete,
 	.xmt_ls_rsp     = lpfc_nvmet_xmt_ls_rsp,
@@ -873,6 +890,7 @@ static struct nvmet_fc_target_template lpfc_tgttemplate = {
 	.fcp_abort      = lpfc_nvmet_xmt_fcp_abort,
 	.fcp_req_release = lpfc_nvmet_xmt_fcp_release,
 	.defer_rcv	= lpfc_nvmet_defer_rcv,
+	.nvme_subsystem_change = lpfc_nvmet_subsystem_change,
 
 	.max_hw_queues  = 1,
 	.max_sgl_segments = LPFC_NVMET_DEFAULT_SEGS,
-- 
2.13.1



From jsmart2021 at gmail.com  Sat Oct 28 10:21:11 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:11 -0700
Subject: [PATCH 2/5] nvmet_fc: support transport subsystem events
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <20171028172114.2045-3-jsmart2021@gmail.com>

This patch adds support for transport events to signal discovery
controller changes.

Add new lldd api for a subsystem change. The lldd is to generate the
appropriate RSCN for the FC-NVME targetport.

Add transport op for discov_chg callback from the nvmet layer.

Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/nvme/target/fc.c       | 10 ++++++++++
 include/linux/nvme-fc-driver.h |  6 ++++++
 2 files changed, 16 insertions(+)

diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c
index 58e010bdda3e..590133c35d11 100644
--- a/drivers/nvme/target/fc.c
+++ b/drivers/nvme/target/fc.c
@@ -2546,6 +2546,15 @@ nvmet_fc_remove_port(struct nvmet_port *port)
 		nvmet_fc_tgtport_put(tgtport);
 }
 
+static void
+nvmet_fc_port_subsys_event(struct nvmet_port *port)
+{
+	struct nvmet_fc_tgtport *tgtport = port->priv;
+
+	if (tgtport && tgtport->ops->nvme_subsystem_change)
+		tgtport->ops->nvme_subsystem_change(&tgtport->fc_target_port);
+}
+
 static struct nvmet_fabrics_ops nvmet_fc_tgt_fcp_ops = {
 	.owner			= THIS_MODULE,
 	.type			= NVMF_TRTYPE_FC,
@@ -2554,6 +2563,7 @@ static struct nvmet_fabrics_ops nvmet_fc_tgt_fcp_ops = {
 	.remove_port		= nvmet_fc_remove_port,
 	.queue_response		= nvmet_fc_fcp_nvme_cmd_done,
 	.delete_ctrl		= nvmet_fc_delete_ctrl,
+	.discov_chg		= nvmet_fc_port_subsys_event,
 };
 
 static int __init nvmet_fc_init_module(void)
diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
index 2be4db353937..ac9d7f190650 100644
--- a/include/linux/nvme-fc-driver.h
+++ b/include/linux/nvme-fc-driver.h
@@ -344,6 +344,11 @@ struct nvme_fc_remote_port {
  *       indicating an FC transport Aborted status.
  *       Entrypoint is Mandatory.
  *
+ * @nvme_subsystem_change:  Called by the transport to generate FC state change
+ *       notifications to NVME initiators. The state change notifications should
+ *       cause the initiator to reconnect to the discovery controller on the
+ *       targetport to look for new discovery log records.
+ *
  * @max_hw_queues:  indicates the maximum number of hw queues the LLDD
  *       supports for cpu affinitization.
  *       Value is Mandatory. Must be at least 1.
@@ -856,6 +861,7 @@ struct nvmet_fc_target_template {
 				struct nvmefc_tgt_fcp_req *fcpreq);
 	void (*defer_rcv)(struct nvmet_fc_target_port *tgtport,
 				struct nvmefc_tgt_fcp_req *fcpreq);
+	void (*nvme_subsystem_change)(struct nvmet_fc_target_port *tgtport);
 
 	u32	max_hw_queues;
 	u16	max_sgl_segments;
-- 
2.13.1



From jsmart2021 at gmail.com  Sat Oct 28 10:21:14 2017
From: jsmart2021 at gmail.com (James Smart)
Date: Sat, 28 Oct 2017 10:21:14 -0700
Subject: [PATCH 5/5] lpfc: Add sysfs interface to post NVME RSCN
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <20171028172114.2045-6-jsmart2021@gmail.com>

From: Dick Kennedy <dick.kennedy at broadcom.com>

To support scenarios which aren't bound to nvmetcli add port scenarios,
which is currently where the transport invokes the nvme_subsystem_changed
callback, add a syfs attribute to lpfc which can be written to cause
an RSCN to be generated for the nport.

Signed-off-by: Dick Kennedy <dick.kennedy at broadcom.com>
Signed-off-by: James Smart <james.smart at broadcom.com>
---
 drivers/scsi/lpfc/lpfc.h      |  1 +
 drivers/scsi/lpfc/lpfc_attr.c | 62 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc.h b/drivers/scsi/lpfc/lpfc.h
index 157788f0bc10..52d403548a3a 100644
--- a/drivers/scsi/lpfc/lpfc.h
+++ b/drivers/scsi/lpfc/lpfc.h
@@ -781,6 +781,7 @@ struct lpfc_hba {
 	uint32_t cfg_use_msi;
 	uint32_t cfg_auto_imax;
 	uint32_t cfg_fcp_imax;
+	uint32_t cfg_force_rscn;
 	uint32_t cfg_fcp_cpu_map;
 	uint32_t cfg_fcp_io_channel;
 	uint32_t cfg_suppress_rsp;
diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c
index c17677f494af..6e8e54e65e25 100644
--- a/drivers/scsi/lpfc/lpfc_attr.c
+++ b/drivers/scsi/lpfc/lpfc_attr.c
@@ -4469,6 +4469,66 @@ static DEVICE_ATTR(lpfc_req_fw_upgrade, S_IRUGO | S_IWUSR,
 		   lpfc_request_firmware_upgrade_store);
 
 /**
+ * lpfc_force_rscn_store
+ *
+ * @dev: class device that is converted into a Scsi_host.
+ * @attr: device attribute, not used.
+ * @buf: unused string
+ * @count: unused variable.
+ *
+ * Description:
+ * Force the switch to send a RSCN to all other NPorts in our zone
+ * If we are direct connect pt2pt, build the RSCN command ourself
+ * and send to the other NPort. Not supported for private loop.
+ *
+ * Returns:
+ * 0      - on success
+ * -EIO   - if command is not sent
+ **/
+static ssize_t
+lpfc_force_rscn_store(struct device *dev, struct device_attribute *attr,
+		      const char *buf, size_t count)
+{
+	struct Scsi_Host *shost = class_to_shost(dev);
+	struct lpfc_vport *vport = (struct lpfc_vport *)shost->hostdata;
+	int i;
+
+	i = lpfc_issue_els_rscn(vport, 0);
+	if (i)
+		return -EIO;
+	return strlen(buf);
+}
+
+/*
+ * lpfc_force_rscn: Force an RSCN to be sent to all remote NPorts
+ * connected to  the HBA.
+ *
+ * Value range is any ascii value
+ */
+static int lpfc_force_rscn;
+module_param(lpfc_force_rscn, int, 0644);
+MODULE_PARM_DESC(lpfc_force_rscn,
+		 "Force an RSCN to be sent to all remote NPorts");
+lpfc_param_show(force_rscn)
+
+/**
+ * lpfc_force_rscn_init - Force an RSCN to be sent to all remote NPorts
+ * @phba: lpfc_hba pointer.
+ * @val: unused value.
+ *
+ * Returns:
+ * zero if val saved.
+ **/
+static int
+lpfc_force_rscn_init(struct lpfc_hba *phba, int val)
+{
+	return 0;
+}
+
+static DEVICE_ATTR(lpfc_force_rscn, 0644,
+		   lpfc_force_rscn_show, lpfc_force_rscn_store);
+
+/**
  * lpfc_fcp_imax_store
  *
  * @dev: class device that is converted into a Scsi_host.
@@ -5218,6 +5278,7 @@ struct device_attribute *lpfc_hba_attrs[] = {
 	&dev_attr_lpfc_nvme_oas,
 	&dev_attr_lpfc_auto_imax,
 	&dev_attr_lpfc_fcp_imax,
+	&dev_attr_lpfc_force_rscn,
 	&dev_attr_lpfc_fcp_cpu_map,
 	&dev_attr_lpfc_fcp_io_channel,
 	&dev_attr_lpfc_suppress_rsp,
@@ -6238,6 +6299,7 @@ lpfc_get_cfgparam(struct lpfc_hba *phba)
 	lpfc_nvme_oas_init(phba, lpfc_nvme_oas);
 	lpfc_auto_imax_init(phba, lpfc_auto_imax);
 	lpfc_fcp_imax_init(phba, lpfc_fcp_imax);
+	lpfc_force_rscn_init(phba, lpfc_force_rscn);
 	lpfc_fcp_cpu_map_init(phba, lpfc_fcp_cpu_map);
 	lpfc_enable_hba_reset_init(phba, lpfc_enable_hba_reset);
 	lpfc_enable_hba_heartbeat_init(phba, lpfc_enable_hba_heartbeat);
-- 
2.13.1



From minwoo.im.dev at gmail.com  Sat Oct 28 11:01:52 2017
From: minwoo.im.dev at gmail.com (Minwoo Im)
Date: Sun, 29 Oct 2017 03:01:52 +0900
Subject: [PATCH] nvme-cli: fix wrong bitmask in number of rpmb units.
Message-ID: <1509213712-32675-1-git-send-email-minwoo.im.dev@gmail.com>

Number of RPMB Units is a 3-bit field[02:00].
Bitmask should be 0111b(7h) to get this field.

Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>
---
 nvme-print.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/nvme-print.c b/nvme-print.c
index 9f524cd..e79b3f4 100644
--- a/nvme-print.c
+++ b/nvme-print.c
@@ -218,7 +218,7 @@ static void show_nvme_id_ctrl_rpmbs(__le32 ctrl_rpmbs)
 	__u32 tsz = (rpmbs & 0xFF0000) >> 16;
 	__u32 rsvd = (rpmbs & 0xFFC0) >> 6;
 	__u32 auth = (rpmbs & 0x38) >> 3;
-	__u32 rpmb = rpmbs & 0x3;
+	__u32 rpmb = rpmbs & 0x7;
 
 	printf(" [31:24]: %#x\tAccess Size\n", asz);
 	printf(" [23:16]: %#x\tTotal Size\n", tsz);
-- 
2.7.4



From minwoo.im.dev at gmail.com  Sat Oct 28 11:26:33 2017
From: minwoo.im.dev at gmail.com (Minwoo Im)
Date: Sun, 29 Oct 2017 03:26:33 +0900
Subject: [PATCH] nvme-cli: make command description const
Message-ID: <1509215193-13119-1-git-send-email-minwoo.im.dev@gmail.com>

Make desc in sanitize_log() const.

Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>
---
 nvme.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/nvme.c b/nvme.c
index 9ac6a3c..ed1c61f 100644
--- a/nvme.c
+++ b/nvme.c
@@ -488,7 +488,7 @@ static const char * sanitize_mon_status_to_string(__u16 status)
 
 static int sanitize_log(int argc, char **argv, struct command *command, struct plugin *plugin)
 {
-	char *desc = "Retrieve sanitize log and show it.";
+	const char *desc = "Retrieve sanitize log and show it.";
 	int fd;
 	int ret;
 	__u8 output[NVME_SANITIZE_LOG_DATA_LEN] = {0};
-- 
2.7.4



From anish.jhaveri at paviliondata.com  Sun Oct 29 01:00:00 2017
From: anish.jhaveri at paviliondata.com (Anish Jhaveri)
Date: Sun, 29 Oct 2017 08:00:00 +0000
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <99f3c781-71ce-f915-d6f4-88d6dd115876@huawei.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de>
 <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>
 <20171028074245.GA15510@lst.de>
 <99f3c781-71ce-f915-d6f4-88d6dd115876@huawei.com>
Message-ID: <DM3PR17MB0825BEF3B830E3ADF08688B097580@DM3PR17MB0825.namprd17.prod.outlook.com>

On Saturday, October 28, 2017 3:10 AM,  Guan Junxiong wrote:

>Does it mean some extra works such as:
>1) showing the path topology of nvme multipath device
Isn't connection to the target suppose guide the host to the shortest and fastest path available path or so called most optimized path. Sysfs entry can easily store that as we store path related info over there. 

>2) daemon to implement immediate and delayed failback
This is also based on target, whenever target is ready for immediate or delayed failback it will let the connect from host succeed. Until then host is in reconnecting state across all path until it finds an optimized or un-optimized path. Why is this extra daemon needed ?
 
>4) grouping paths besides ANA
Why we cannot use NS Group here.

Would be good idea to move away from legacy way of doing things for NVME devices. NVME Multipath implementation by Christoph is very simple. How about not making it super complicated.  

Best regards,
Anish



From hch at lst.de  Sun Oct 29 01:44:26 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:26 +0200
Subject: controller deletion consolidation
Message-ID: <20171029084431.8702-1-hch@lst.de>

Hi all,

this series consolidates a lot of the boilerplate code in controller
deletion.  I did this when applying the FC patches that finally made
it look exactly like RDMA.

This will probably conflict a bit with Sagis consolidation series, but
in the end should make it easier.

Two notes:

 - do we need the cancellation of reset_work in FC?  If so we probably
   want it in RDMA and loop as well.
 - should RDMA really queue delete_work manually in
   nvme_rdma_reconnect_or_remove instead of going through the state
   machine?  That seems like a bug to me.


From hch at lst.de  Sun Oct 29 01:44:27 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:27 +0200
Subject: [PATCH 1/5] nvme-fc: avoid workqueue flush stalls
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <20171029084431.8702-2-hch@lst.de>

From: James Smart <jsmart2021 at gmail.com>

There's no need to wait for the full nvme_wq, which is now shared,
to flush. flush only the delete_work item.

Signed-off-by: James Smart <james.smart at broadcom.com>
Reviewed-by: Sagi Grimberg <sgi at grimberg.me>
Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/fc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index b600c07803cf..50cc17e99475 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2693,7 +2693,7 @@ nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
 	nvme_get_ctrl(&ctrl->ctrl);
 	ret = __nvme_fc_del_ctrl(ctrl);
 	if (!ret)
-		flush_workqueue(nvme_wq);
+		flush_work(&ctrl->delete_work);
 	nvme_put_ctrl(&ctrl->ctrl);
 
 	return ret;
-- 
2.14.2



From hch at lst.de  Sun Oct 29 01:44:28 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:28 +0200
Subject: [PATCH 2/5] nvme-fc: merge __nvme_fc_schedule_delete_work into
 __nvme_fc_del_ctrl
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <20171029084431.8702-3-hch@lst.de>

No need to have two functions doing the same thing.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/fc.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 50cc17e99475..827e9efbe556 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2663,22 +2663,14 @@ nvme_fc_delete_ctrl_work(struct work_struct *work)
 	nvme_put_ctrl(&ctrl->ctrl);
 }
 
-static bool
-__nvme_fc_schedule_delete_work(struct nvme_fc_ctrl *ctrl)
-{
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
-		return true;
-
-	if (!queue_work(nvme_wq, &ctrl->delete_work))
-		return true;
-
-	return false;
-}
-
 static int
 __nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
 {
-	return __nvme_fc_schedule_delete_work(ctrl) ? -EBUSY : 0;
+	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
+		return -EBUSY;
+	if (!queue_work(nvme_wq, &ctrl->delete_work))
+		return -EBUSY;
+	return 0;
 }
 
 /*
@@ -2724,7 +2716,7 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 				"NVME-FC{%d}: Max reconnect attempts (%d) "
 				"reached. Removing controller\n",
 				ctrl->cnum, ctrl->ctrl.nr_reconnects);
-		WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
+		WARN_ON(__nvme_fc_del_ctrl(ctrl));
 	}
 }
 
-- 
2.14.2



From hch at lst.de  Sun Oct 29 01:44:29 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:29 +0200
Subject: [PATCH 3/5] nvme: move controller deletion to common code
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <20171029084431.8702-4-hch@lst.de>

Move the ->delete_work and the associated helpers to common code instead
of duplicating them in every driver.  This also adds the missing reference
get/put for the loop driver.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c    | 38 ++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/fabrics.c |  2 +-
 drivers/nvme/host/fc.c      | 42 +++++----------------------------------
 drivers/nvme/host/nvme.h    |  5 ++++-
 drivers/nvme/host/rdma.c    | 45 ++++++------------------------------------
 drivers/nvme/target/loop.c  | 48 +++++++++------------------------------------
 6 files changed, 62 insertions(+), 118 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index df525ab42fcd..d835ac05bbf7 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -97,6 +97,41 @@ static int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
 	return ret;
 }
 
+static void nvme_delete_ctrl_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, delete_work);
+
+	ctrl->ops->delete_ctrl(ctrl);
+}
+
+int nvme_delete_ctrl(struct nvme_ctrl *ctrl)
+{
+	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_DELETING))
+		return -EBUSY;
+	if (!queue_work(nvme_wq, &ctrl->delete_work))
+		return -EBUSY;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_delete_ctrl);
+
+int nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl)
+{
+	int ret = 0;
+
+	/*
+	 * Keep a reference until the work is flushed since ->delete_ctrl
+	 * can free the controller.
+	 */
+	nvme_get_ctrl(ctrl);
+	ret = nvme_delete_ctrl(ctrl);
+	if (!ret)
+		flush_work(&ctrl->delete_work);
+	nvme_put_ctrl(ctrl);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_delete_ctrl_sync);
+
 static blk_status_t nvme_error_status(struct request *req)
 {
 	switch (nvme_req(req)->status & 0x7ff) {
@@ -2122,7 +2157,7 @@ static ssize_t nvme_sysfs_delete(struct device *dev,
 	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
 
 	if (device_remove_file_self(dev, attr))
-		ctrl->ops->delete_ctrl(ctrl);
+		nvme_delete_ctrl_sync(ctrl);
 	return count;
 }
 static DEVICE_ATTR(delete_controller, S_IWUSR, NULL, nvme_sysfs_delete);
@@ -2702,6 +2737,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 	INIT_WORK(&ctrl->scan_work, nvme_scan_work);
 	INIT_WORK(&ctrl->async_event_work, nvme_async_event_work);
 	INIT_WORK(&ctrl->fw_act_work, nvme_fw_act_work);
+	INIT_WORK(&ctrl->delete_work, nvme_delete_ctrl_work);
 
 	ret = ida_simple_get(&nvme_instance_ida, 0, 0, GFP_KERNEL);
 	if (ret < 0)
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 4a83137b0268..a4967de5bb25 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -887,7 +887,7 @@ nvmf_create_ctrl(struct device *dev, const char *buf, size_t count)
 			"controller returned incorrect NQN: \"%s\".\n",
 			ctrl->subnqn);
 		up_read(&nvmf_transports_rwsem);
-		ctrl->ops->delete_ctrl(ctrl);
+		nvme_delete_ctrl_sync(ctrl);
 		return ERR_PTR(-EINVAL);
 	}
 
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 827e9efbe556..a7bdb17de29d 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -157,7 +157,6 @@ struct nvme_fc_ctrl {
 	struct blk_mq_tag_set	admin_tag_set;
 	struct blk_mq_tag_set	tag_set;
 
-	struct work_struct	delete_work;
 	struct delayed_work	connect_work;
 
 	struct kref		ref;
@@ -223,7 +222,6 @@ static struct device *fc_udev_device;
 
 /* *********************** FC-NVME Port Management ************************ */
 
-static int __nvme_fc_del_ctrl(struct nvme_fc_ctrl *);
 static void __nvme_fc_delete_hw_queue(struct nvme_fc_ctrl *,
 			struct nvme_fc_queue *, unsigned int);
 
@@ -662,7 +660,7 @@ nvme_fc_unregister_remoteport(struct nvme_fc_remote_port *portptr)
 
 	/* tear down all associations to the remote port */
 	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list)
-		__nvme_fc_del_ctrl(ctrl);
+		nvme_delete_ctrl(&ctrl->ctrl);
 
 	spin_unlock_irqrestore(&rport->lock, flags);
 
@@ -2636,10 +2634,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
 }
 
 static void
-nvme_fc_delete_ctrl_work(struct work_struct *work)
+nvme_fc_delete_ctrl(struct nvme_ctrl *nctrl)
 {
-	struct nvme_fc_ctrl *ctrl =
-		container_of(work, struct nvme_fc_ctrl, delete_work);
+	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
 
 	cancel_work_sync(&ctrl->ctrl.reset_work);
 	cancel_delayed_work_sync(&ctrl->connect_work);
@@ -2663,34 +2660,6 @@ nvme_fc_delete_ctrl_work(struct work_struct *work)
 	nvme_put_ctrl(&ctrl->ctrl);
 }
 
-static int
-__nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
-{
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
-		return -EBUSY;
-	if (!queue_work(nvme_wq, &ctrl->delete_work))
-		return -EBUSY;
-	return 0;
-}
-
-/*
- * Request from nvme core layer to delete the controller
- */
-static int
-nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
-{
-	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
-	int ret;
-
-	nvme_get_ctrl(&ctrl->ctrl);
-	ret = __nvme_fc_del_ctrl(ctrl);
-	if (!ret)
-		flush_work(&ctrl->delete_work);
-	nvme_put_ctrl(&ctrl->ctrl);
-
-	return ret;
-}
-
 static void
 nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 {
@@ -2716,7 +2685,7 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
 				"NVME-FC{%d}: Max reconnect attempts (%d) "
 				"reached. Removing controller\n",
 				ctrl->cnum, ctrl->ctrl.nr_reconnects);
-		WARN_ON(__nvme_fc_del_ctrl(ctrl));
+		WARN_ON(nvme_delete_ctrl(&ctrl->ctrl));
 	}
 }
 
@@ -2748,7 +2717,7 @@ static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_fc_nvme_ctrl_freed,
 	.submit_async_event	= nvme_fc_submit_async_event,
-	.delete_ctrl		= nvme_fc_del_nvme_ctrl,
+	.delete_ctrl		= nvme_fc_delete_ctrl,
 	.get_address		= nvmf_get_address,
 	.reinit_request		= nvme_fc_reinit_request,
 };
@@ -2851,7 +2820,6 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	get_device(ctrl->dev);
 	kref_init(&ctrl->ref);
 
-	INIT_WORK(&ctrl->delete_work, nvme_fc_delete_ctrl_work);
 	INIT_WORK(&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);
 	INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
 	spin_lock_init(&ctrl->lock);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1bb2bc165e54..35d9cee515f1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -138,6 +138,7 @@ struct nvme_ctrl {
 	struct cdev cdev;
 	struct ida ns_ida;
 	struct work_struct reset_work;
+	struct work_struct delete_work;
 
 	struct opal_dev *opal_dev;
 
@@ -236,7 +237,7 @@ struct nvme_ctrl_ops {
 	int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 	void (*free_ctrl)(struct nvme_ctrl *ctrl);
 	void (*submit_async_event)(struct nvme_ctrl *ctrl, int aer_idx);
-	int (*delete_ctrl)(struct nvme_ctrl *ctrl);
+	void (*delete_ctrl)(struct nvme_ctrl *ctrl);
 	int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
 	int (*reinit_request)(void *data, struct request *rq);
 };
@@ -339,6 +340,8 @@ int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count);
 void nvme_start_keep_alive(struct nvme_ctrl *ctrl);
 void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);
 int nvme_reset_ctrl(struct nvme_ctrl *ctrl);
+int nvme_delete_ctrl(struct nvme_ctrl *ctrl);
+int nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl);
 
 #ifdef CONFIG_NVM
 int nvme_nvm_register(struct nvme_ns *ns, char *disk_name, int node);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 32b0a9ef26e6..5175b465997d 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -105,7 +105,6 @@ struct nvme_rdma_ctrl {
 
 	/* other member variables */
 	struct blk_mq_tag_set	tag_set;
-	struct work_struct	delete_work;
 	struct work_struct	err_work;
 
 	struct nvme_rdma_qe	async_event_sqe;
@@ -913,7 +912,7 @@ static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl)
 				ctrl->ctrl.opts->reconnect_delay * HZ);
 	} else {
 		dev_info(ctrl->ctrl.device, "Removing controller...\n");
-		queue_work(nvme_wq, &ctrl->delete_work);
+		queue_work(nvme_wq, &ctrl->ctrl.delete_work);
 	}
 }
 
@@ -1764,41 +1763,10 @@ static void nvme_rdma_remove_ctrl(struct nvme_rdma_ctrl *ctrl)
 	nvme_put_ctrl(&ctrl->ctrl);
 }
 
-static void nvme_rdma_del_ctrl_work(struct work_struct *work)
+static void nvme_rdma_delete_ctrl(struct nvme_ctrl *ctrl)
 {
-	struct nvme_rdma_ctrl *ctrl = container_of(work,
-				struct nvme_rdma_ctrl, delete_work);
-
-	nvme_stop_ctrl(&ctrl->ctrl);
-	nvme_rdma_remove_ctrl(ctrl);
-}
-
-static int __nvme_rdma_del_ctrl(struct nvme_rdma_ctrl *ctrl)
-{
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
-		return -EBUSY;
-
-	if (!queue_work(nvme_wq, &ctrl->delete_work))
-		return -EBUSY;
-
-	return 0;
-}
-
-static int nvme_rdma_del_ctrl(struct nvme_ctrl *nctrl)
-{
-	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
-	int ret = 0;
-
-	/*
-	 * Keep a reference until all work is flushed since
-	 * __nvme_rdma_del_ctrl can free the ctrl mem
-	 */
-	nvme_get_ctrl(&ctrl->ctrl);
-	ret = __nvme_rdma_del_ctrl(ctrl);
-	if (!ret)
-		flush_work(&ctrl->delete_work);
-	nvme_put_ctrl(&ctrl->ctrl);
-	return ret;
+	nvme_stop_ctrl(ctrl);
+	nvme_rdma_remove_ctrl(to_rdma_ctrl(ctrl));
 }
 
 static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
@@ -1846,7 +1814,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_rdma_free_ctrl,
 	.submit_async_event	= nvme_rdma_submit_async_event,
-	.delete_ctrl		= nvme_rdma_del_ctrl,
+	.delete_ctrl		= nvme_rdma_delete_ctrl,
 	.get_address		= nvmf_get_address,
 	.reinit_request		= nvme_rdma_reinit_request,
 };
@@ -1977,7 +1945,6 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	INIT_DELAYED_WORK(&ctrl->reconnect_work,
 			nvme_rdma_reconnect_ctrl_work);
 	INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work);
-	INIT_WORK(&ctrl->delete_work, nvme_rdma_del_ctrl_work);
 	INIT_WORK(&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work);
 
 	ctrl->ctrl.queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */
@@ -2081,7 +2048,7 @@ static void nvme_rdma_remove_one(struct ib_device *ib_device, void *client_data)
 		dev_info(ctrl->ctrl.device,
 			"Removing ctrl: NQN \"%s\", addr %pISp\n",
 			ctrl->ctrl.opts->subsysnqn, &ctrl->addr);
-		__nvme_rdma_del_ctrl(ctrl);
+		nvme_delete_ctrl(&ctrl->ctrl);
 	}
 	mutex_unlock(&nvme_rdma_ctrl_mutex);
 
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index f83e925fe64a..7f9f3fc3fb2a 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -53,7 +53,6 @@ struct nvme_loop_ctrl {
 	struct nvme_ctrl	ctrl;
 
 	struct nvmet_ctrl	*target_ctrl;
-	struct work_struct	delete_work;
 };
 
 static inline struct nvme_loop_ctrl *to_loop_ctrl(struct nvme_ctrl *ctrl)
@@ -439,41 +438,13 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)
 	nvme_loop_destroy_admin_queue(ctrl);
 }
 
-static void nvme_loop_del_ctrl_work(struct work_struct *work)
+static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl)
 {
-	struct nvme_loop_ctrl *ctrl = container_of(work,
-				struct nvme_loop_ctrl, delete_work);
-
-	nvme_stop_ctrl(&ctrl->ctrl);
-	nvme_remove_namespaces(&ctrl->ctrl);
-	nvme_loop_shutdown_ctrl(ctrl);
-	nvme_uninit_ctrl(&ctrl->ctrl);
-	nvme_put_ctrl(&ctrl->ctrl);
-}
-
-static int __nvme_loop_del_ctrl(struct nvme_loop_ctrl *ctrl)
-{
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
-		return -EBUSY;
-
-	if (!queue_work(nvme_wq, &ctrl->delete_work))
-		return -EBUSY;
-
-	return 0;
-}
-
-static int nvme_loop_del_ctrl(struct nvme_ctrl *nctrl)
-{
-	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(nctrl);
-	int ret;
-
-	ret = __nvme_loop_del_ctrl(ctrl);
-	if (ret)
-		return ret;
-
-	flush_work(&ctrl->delete_work);
-
-	return 0;
+	nvme_stop_ctrl(ctrl);
+	nvme_remove_namespaces(ctrl);
+	nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl));
+	nvme_uninit_ctrl(ctrl);
+	nvme_put_ctrl(ctrl);
 }
 
 static void nvme_loop_delete_ctrl(struct nvmet_ctrl *nctrl)
@@ -483,7 +454,7 @@ static void nvme_loop_delete_ctrl(struct nvmet_ctrl *nctrl)
 	mutex_lock(&nvme_loop_ctrl_mutex);
 	list_for_each_entry(ctrl, &nvme_loop_ctrl_list, list) {
 		if (ctrl->ctrl.cntlid == nctrl->cntlid)
-			__nvme_loop_del_ctrl(ctrl);
+			nvme_delete_ctrl(&ctrl->ctrl);
 	}
 	mutex_unlock(&nvme_loop_ctrl_mutex);
 }
@@ -539,7 +510,7 @@ static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_loop_free_ctrl,
 	.submit_async_event	= nvme_loop_submit_async_event,
-	.delete_ctrl		= nvme_loop_del_ctrl,
+	.delete_ctrl		= nvme_loop_delete_ctrl_host,
 };
 
 static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)
@@ -601,7 +572,6 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 	ctrl->ctrl.opts = opts;
 	INIT_LIST_HEAD(&ctrl->list);
 
-	INIT_WORK(&ctrl->delete_work, nvme_loop_del_ctrl_work);
 	INIT_WORK(&ctrl->ctrl.reset_work, nvme_loop_reset_ctrl_work);
 
 	ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_loop_ctrl_ops,
@@ -731,7 +701,7 @@ static void __exit nvme_loop_cleanup_module(void)
 
 	mutex_lock(&nvme_loop_ctrl_mutex);
 	list_for_each_entry_safe(ctrl, next, &nvme_loop_ctrl_list, list)
-		__nvme_loop_del_ctrl(ctrl);
+		nvme_delete_ctrl(&ctrl->ctrl);
 	mutex_unlock(&nvme_loop_ctrl_mutex);
 
 	flush_workqueue(nvme_wq);
-- 
2.14.2



From hch at lst.de  Sun Oct 29 01:44:30 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:30 +0200
Subject: [PATCH 4/5] nvme-rdma: remove nvme_rdma_remove_ctrl
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <20171029084431.8702-5-hch@lst.de>

It is only used in two places, and some of the work done by it will
be taken into common code soon.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/rdma.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 5175b465997d..a3521b852ea8 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1755,18 +1755,13 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 	nvme_rdma_destroy_admin_queue(ctrl, shutdown);
 }
 
-static void nvme_rdma_remove_ctrl(struct nvme_rdma_ctrl *ctrl)
-{
-	nvme_remove_namespaces(&ctrl->ctrl);
-	nvme_rdma_shutdown_ctrl(ctrl, true);
-	nvme_uninit_ctrl(&ctrl->ctrl);
-	nvme_put_ctrl(&ctrl->ctrl);
-}
-
 static void nvme_rdma_delete_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_stop_ctrl(ctrl);
-	nvme_rdma_remove_ctrl(to_rdma_ctrl(ctrl));
+	nvme_remove_namespaces(ctrl);
+	nvme_rdma_shutdown_ctrl(to_rdma_ctrl(ctrl), true);
+	nvme_uninit_ctrl(ctrl);
+	nvme_put_ctrl(ctrl);
 }
 
 static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
@@ -1802,7 +1797,10 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
 
 out_fail:
 	dev_warn(ctrl->ctrl.device, "Removing after reset failure\n");
-	nvme_rdma_remove_ctrl(ctrl);
+	nvme_remove_namespaces(&ctrl->ctrl);
+	nvme_rdma_shutdown_ctrl(ctrl, true);
+	nvme_uninit_ctrl(&ctrl->ctrl);
+	nvme_put_ctrl(&ctrl->ctrl);
 }
 
 static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
-- 
2.14.2



From hch at lst.de  Sun Oct 29 01:44:31 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 10:44:31 +0200
Subject: [PATCH 5/5] nvme: consolidate common code from ->reset_work
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <20171029084431.8702-6-hch@lst.de>

No change in behavior except that the FC code cancels two work items a
little later now.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/core.c   |  4 ++++
 drivers/nvme/host/fc.c     | 13 -------------
 drivers/nvme/host/rdma.c   |  4 ----
 drivers/nvme/target/loop.c |  4 ----
 4 files changed, 4 insertions(+), 21 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d835ac05bbf7..4fa748c9a3f6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -102,7 +102,11 @@ static void nvme_delete_ctrl_work(struct work_struct *work)
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, delete_work);
 
+	nvme_stop_ctrl(ctrl);
+	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
+	nvme_uninit_ctrl(ctrl);
+	nvme_put_ctrl(ctrl);
 }
 
 int nvme_delete_ctrl(struct nvme_ctrl *ctrl)
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index a7bdb17de29d..e447b532b9ee 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2640,24 +2640,11 @@ nvme_fc_delete_ctrl(struct nvme_ctrl *nctrl)
 
 	cancel_work_sync(&ctrl->ctrl.reset_work);
 	cancel_delayed_work_sync(&ctrl->connect_work);
-	nvme_stop_ctrl(&ctrl->ctrl);
-	nvme_remove_namespaces(&ctrl->ctrl);
 	/*
 	 * kill the association on the link side.  this will block
 	 * waiting for io to terminate
 	 */
 	nvme_fc_delete_association(ctrl);
-
-	/*
-	 * tear down the controller
-	 * After the last reference on the nvme ctrl is removed,
-	 * the transport nvme_fc_nvme_ctrl_freed() callback will be
-	 * invoked. From there, the transport will tear down it's
-	 * logical queues and association.
-	 */
-	nvme_uninit_ctrl(&ctrl->ctrl);
-
-	nvme_put_ctrl(&ctrl->ctrl);
 }
 
 static void
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index a3521b852ea8..ed6e05018a92 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1757,11 +1757,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 
 static void nvme_rdma_delete_ctrl(struct nvme_ctrl *ctrl)
 {
-	nvme_stop_ctrl(ctrl);
-	nvme_remove_namespaces(ctrl);
 	nvme_rdma_shutdown_ctrl(to_rdma_ctrl(ctrl), true);
-	nvme_uninit_ctrl(ctrl);
-	nvme_put_ctrl(ctrl);
 }
 
 static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 7f9f3fc3fb2a..bc95c6ed531a 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -440,11 +440,7 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)
 
 static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl)
 {
-	nvme_stop_ctrl(ctrl);
-	nvme_remove_namespaces(ctrl);
 	nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl));
-	nvme_uninit_ctrl(ctrl);
-	nvme_put_ctrl(ctrl);
 }
 
 static void nvme_loop_delete_ctrl(struct nvmet_ctrl *nctrl)
-- 
2.14.2



From hch at lst.de  Sun Oct 29 01:57:52 2017
From: hch at lst.de (Christoph Hellwig)
Date: Sun, 29 Oct 2017 09:57:52 +0100
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <99f3c781-71ce-f915-d6f4-88d6dd115876@huawei.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de>
 <dc988a8a-8c88-fe20-34a0-de5cdd615b6b@huawei.com>
 <20171028074245.GA15510@lst.de>
 <99f3c781-71ce-f915-d6f4-88d6dd115876@huawei.com>
Message-ID: <20171029085751.GA8793@lst.de>

On Sat, Oct 28, 2017 at 06:09:46PM +0800, Guan Junxiong wrote:
> Does it mean some extra works such as:
> 1) showing the path topology of nvme multipath device

It's in sysfs.  And Johannes volunteered to also add nvme-cli
support.

> 2) daemon to implement immediate and delayed failback

The whole point is to not have a daemon.

> 3) detecting sub-healthy path due to shaky link

We can do this in kernel space.  It just needs someone to implement it.

> 4) grouping paths besides ANA

We don't want to do non-standard grouping.  Please work with the
NVMe working group for your grouping ideas.


From hare at suse.de  Sun Oct 29 03:01:54 2017
From: hare at suse.de (Hannes Reinecke)
Date: Sun, 29 Oct 2017 11:01:54 +0100
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <20171028141745.GA32039@redhat.com>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de> <20171028141745.GA32039@redhat.com>
Message-ID: <ae4180e2-4c1b-a5f0-5c26-7cec692e6d50@suse.de>

On 10/28/2017 04:17 PM, Mike Snitzer wrote:
> On Sat, Oct 28 2017 at  2:38am -0400,
> Christoph Hellwig <hch at lst.de> wrote:
> 
>> On Tue, Oct 24, 2017 at 05:40:00PM -0400, Mike Snitzer wrote:
[ .. ]
>>> Ah well.  There is only one correct way to do NVMe multipathing after
>>> all right?
>>
>> I don't think you'll get very useful results, even if you try.  But I
>> guess we'll just have to tell people to use SuSE if they want NVMe
>> multipathing to work then :)
> 
> Don't do that.  Don't assume you know it all.  Don't fabricate vendor
> wars in your head and then project it out to the rest of the community.
> We're all in this together.
> 
> Fact is Hannes and I have exchanged private mail (in response to this
> very thread) and we agree that your approach is currently not suitable
> for enterprise deployment.  Hannes needs to also deal with the coming
> duality of Linux multipathing and you aren't making it easy.  Just
> because you're able to be myopic doesn't mean the rest of us with way
> more direct customers behind us can be.
> 
> You're finding your way with this new multipath model and I'm happy to
> see that happen.  But what I'm not happy about is the steps you're
> taking to be actively disruptive.  There were so many ways this all
> could've gone and sadly you've elected to stand up an architecture that
> doesn't even allow the prospect of reuse.  And for what?  Because doing
> so takes 10% more work?  Well we can backfill that work if you'll grant
> us an open-mind.
> 
> I'm really not against you.  I just need very basic controls put into
> the NVMe multipathing code that allows it to be disabled (yet reused).
> Not that I have immediate plans to actually _use_ it.  My hope is I can
> delegate NVMe multipathing to NVMe!  But I need the latitude to find my
> way with the requirements I am, or will be, needing to consider.
> 
> Please don't paint me and others in my position into a corner.
> 
To add my some cents from the SUSE perspective:
We have _quite_ some deployments on dm-multipathing, and most of our
customers are quite accustomed to setting up deployments with that.
It will be impossible to ensure that all customers and installation
scripts will _not_ try starting up multipathing, and for some scenarios
even preferring to use dm-multipath just because their tooling is geared
up for that.
So we absolutely need peaceful coexistence between dm-multipathing and
nvme multipathing. The precise level can be discussed (whether it being
a global on/off switch or some more fine-grained thingie), but we simply
cannot declare nvme multipathing being the one true way for NVMe.
The support-calls alone will kill us.

Note: this has _nothing_ to do with performance. I'm perfectly fine to
accept that dm-multipath has sub-optimal performance.
But that should not imply that it cannot be used for NVMe.

After all, Linux is about choice, not about forcing users to do things
in one way only.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)


From sagi at grimberg.me  Sun Oct 29 04:57:45 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 13:57:45 +0200
Subject: controller deletion consolidation
In-Reply-To: <20171029084431.8702-1-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
Message-ID: <c82f8c4f-052a-7233-d0df-59c7c6a7a1a6@grimberg.me>


> Hi all,
> 
> this series consolidates a lot of the boilerplate code in controller
> deletion.  I did this when applying the FC patches that finally made
> it look exactly like RDMA.
> 
> This will probably conflict a bit with Sagis consolidation series, but
> in the end should make it easier.

Definitely a step in the right direction. Looks good.

> Two notes:
> 
>   - do we need the cancellation of reset_work in FC?  If so we probably
>     want it in RDMA and loop as well.

We probably do.

>   - should RDMA really queue delete_work manually in
>     nvme_rdma_reconnect_or_remove instead of going through the state
>     machine?  That seems like a bug to me.
> 

I had this fix from the centralization patches, I can easily extract it
out and send it asap.


From sagi at grimberg.me  Sun Oct 29 05:21:01 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 14:21:01 +0200
Subject: [PATCH 1/2] nvme-rdma: reuse nvme_delete_ctrl when reconnect attempts
 expire
Message-ID: <1509279662-32169-1-git-send-email-sagi@grimberg.me>

instead of just queueing delete work

Reported-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 4f8a249a2c12..5e95a9860f35 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -912,7 +912,7 @@ static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl)
 				ctrl->ctrl.opts->reconnect_delay * HZ);
 	} else {
 		dev_info(ctrl->ctrl.device, "Removing controller...\n");
-		queue_work(nvme_wq, &ctrl->ctrl.delete_work);
+		nvme_delete_ctrl(&ctrl->ctrl);
 	}
 }
 
-- 
2.7.4



From sagi at grimberg.me  Sun Oct 29 05:21:02 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 14:21:02 +0200
Subject: [PATCH 2/2] nvme-core: flush reset_work before safely continuing with
 delete operation
In-Reply-To: <1509279662-32169-1-git-send-email-sagi@grimberg.me>
References: <1509279662-32169-1-git-send-email-sagi@grimberg.me>
Message-ID: <1509279662-32169-2-git-send-email-sagi@grimberg.me>

Prevent racing controller reset and delete flows. reset_work must not
ever self-requeue so flushing it suffices.

Reported-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 1 +
 drivers/nvme/host/fc.c   | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 05161b700b59..d495c1d1f11c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -102,6 +102,7 @@ static void nvme_delete_ctrl_work(struct work_struct *work)
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, delete_work);
 
+	flush_work(&ctrl->reset_work);
 	nvme_stop_ctrl(ctrl);
 	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 780060c48218..cc214f5a3480 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2655,7 +2655,6 @@ nvme_fc_delete_ctrl(struct nvme_ctrl *nctrl)
 {
 	struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
 
-	cancel_work_sync(&ctrl->ctrl.reset_work);
 	cancel_delayed_work_sync(&ctrl->connect_work);
 	/*
 	 * kill the association on the link side.  this will block
-- 
2.7.4



From sagi at grimberg.me  Sun Oct 29 05:24:34 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 14:24:34 +0200
Subject: [PATCH v4] nvmet: Fix fatal_err_work deadlock
In-Reply-To: <20171027201249.31728-1-jsmart2021@gmail.com>
References: <20171027201249.31728-1-jsmart2021@gmail.com>
Message-ID: <8e8ab4ca-dec8-60e5-f327-217022f1d3c0@grimberg.me>

Looks good James,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Sun Oct 29 05:25:10 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 14:25:10 +0200
Subject: [PATCH urgent-4.14] nvme: Fix setting logical block format when
 revalidating
In-Reply-To: <20171027195122.1810-1-keith.busch@intel.com>
References: <20171027195122.1810-1-keith.busch@intel.com>
Message-ID: <bc217aee-a711-357a-d1f3-bf4479ecbd64@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Sun Oct 29 05:28:31 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 14:28:31 +0200
Subject: [PATCH 1/2] nvme_fc: fix localport resume using stale values
In-Reply-To: <20171027172805.13495-2-jsmart2021@gmail.com>
References: <20171027172805.13495-1-jsmart2021@gmail.com>
 <20171027172805.13495-2-jsmart2021@gmail.com>
Message-ID: <4640f307-5d61-838b-62d4-c8310ff4bd3b@grimberg.me>


> The localport resume was not updating the lldd ops structure. If the
> lldd is unloaded and reloaded, the ops pointers will differ.

Question, why is the lport kept around if the lldd unloaded?


From sagi at grimberg.me  Sun Oct 29 09:11:55 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 18:11:55 +0200
Subject: [PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller
 rescan
In-Reply-To: <20171028172114.2045-1-jsmart2021@gmail.com>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
Message-ID: <286e4dc4-af2c-3482-4345-a7b9984da994@grimberg.me>

James,

> A transport may have a transport-specific mechanism that can signal
> when discovery controller content has changed and request a host
> to reconnect to the discovery controller.
> 
> FC is such a transport. RSCNs may be generated by the FC port with
> the discovery server, with the RSCNs then broadcast to the FC-NVME
> hosts. A host, upon receiving the RSCN, would validate connectivity
> then initiate a discovery controller rescan, allowing new subsystems
> to be connected to or updating subsystem connectivity tables.

How does this fit with possible future discovery enhancements discussed
in nvme TWG right now? The notification for discovery records change
will not be transport specific in the future.


From idanb at mellanox.com  Sun Oct 29 09:38:21 2017
From: idanb at mellanox.com (idanb at mellanox.com)
Date: Sun, 29 Oct 2017 18:38:21 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
Message-ID: <1509295101-14081-3-git-send-email-idanb@mellanox.com>

From: Idan Burstein <idanb at mellanox.com>

NVMe over Fabrics in its secure "register_always" mode
registers and invalidates the user buffer upon each IO.
The protocol enables the host to request the susbsystem
to use SEND WITH INVALIDATE operation while returning the
response capsule and invalidate the local key
(remote_invalidation).
In some HW implementations, the local network adapter may
perform better while using local invalidation operations.

The results below show that running with local invalidation
rather then remote invalidation improve the iops one could
achieve by using the ConnectX-5Ex network adapter by x1.36 factor.
Nevertheless, using local invalidation induce more CPU overhead
than enabling the target to invalidate remotly, therefore,
because there is a CPU% vs IOPs limit tradeoff we propose to
have module parameter to define whether to request remote
invalidation or not.

The following results were taken against a single nvme over fabrics
subsystem with a single namespace backed by null_blk:

Block Size       s/g reg_wr      inline reg_wr    inline reg_wr + local inv
++++++++++++   ++++++++++++++   ++++++++++++++++ +++++++++++++++++++++++++++
512B            1446.6K/8.57%    5224.2K/76.21%   7143.3K/79.72%
1KB             1390.6K/8.5%     4656.7K/71.69%   5860.6K/55.45%
2KB             1343.8K/8.6%     3410.3K/38.96%   4106.7K/55.82%
4KB             1254.8K/8.39%    2033.6K/15.86%   2165.3K/17.48%
8KB             1079.5K/7.7%     1143.1K/7.27%    1158.2K/7.33%
16KB            603.4K/3.64%     593.8K/3.4%      588.9K/3.77%
32KB            294.8K/2.04%     293.7K/1.98%     294.4K/2.93%
64KB            138.2K/1.32%     141.6K/1.26%     135.6K/1.34%

Signed-off-by: Max Gurtovoy <maxg at mellanox.com>
Signed-off-by: Idan Burstein <idanb at mellanox.com>
---
 drivers/nvme/host/rdma.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff..7f8225d 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -146,6 +146,11 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl)
 MODULE_PARM_DESC(register_always,
 	 "Use memory registration even for contiguous memory regions");
 
+static bool remote_invalidation = true;
+module_param(remote_invalidation, bool, 0444);
+MODULE_PARM_DESC(remote_invalidation,
+	 "request remote invalidation from subsystem (default: true)");
+
 static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
 		struct rdma_cm_event *event);
 static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
@@ -1152,8 +1157,9 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
 	sg->addr = cpu_to_le64(req->mr->iova);
 	put_unaligned_le24(req->mr->length, sg->length);
 	put_unaligned_le32(req->mr->rkey, sg->key);
-	sg->type = (NVME_KEY_SGL_FMT_DATA_DESC << 4) |
-			NVME_SGL_FMT_INVALIDATE;
+	sg->type = NVME_KEY_SGL_FMT_DATA_DESC << 4;
+	if (remote_invalidation)
+		sg->type |= NVME_SGL_FMT_INVALIDATE;
 
 	return 0;
 }
-- 
1.8.3.1



From idanb at mellanox.com  Sun Oct 29 09:38:19 2017
From: idanb at mellanox.com (idanb at mellanox.com)
Date: Sun, 29 Oct 2017 18:38:19 +0200
Subject: [PATCH 0/2] Performance Improvents for Secured Mode NVMe over Fabrics
 and other RDMA ULPs 
Message-ID: <1509295101-14081-1-git-send-email-idanb@mellanox.com>

From: Idan Burstein <idanb at mellanox.com>

The following two patches are including performance 
optimizations for RDMA kernel ULPs that register/invalidate
per IO (e.g. NVMe over Fabrics, iSER, NFS RDMA). The patches 
improve the IOPs could be achieved by a single network 
adapter with the above ULPs, specifically for NVMe over 
Fabrics the IOPs have improved x5 for small IO reads could 
be achieved in Linux using a single network adapter.

First improvement, which is relavant to mlx5 rdma adapters, 
improves the way reg_wr is posted to the adapter by communicating 
the KLM/MTT list inline (within the work request) instead of using 
scatter-gather.

After the first bottleneck has been removed, a new bottleneck 
was uncovered for send with invaldate processing in the recieve 
path of the ConnectX-5 (mlx5) adapter, therefore a new module 
parameter was added for nvme_rdma for disabling remote 
invalidation in such case.

We are currently investigating write IOPs optimizations as
well, be tuned for more patches in the pipeline...

Idan Burstein (2):
  IB/mlx5: posting klm/mtt list inline in the send queue for reg_wr
  nvme-rdma: Add remote_invalidation module parameter

 drivers/infiniband/hw/mlx5/qp.c | 41 +++++++++++++++++++++++++++++++++++------
 drivers/nvme/host/rdma.c        | 10 ++++++++--
 2 files changed, 43 insertions(+), 8 deletions(-)

-- 
1.8.3.1



From idanb at mellanox.com  Sun Oct 29 09:38:20 2017
From: idanb at mellanox.com (idanb at mellanox.com)
Date: Sun, 29 Oct 2017 18:38:20 +0200
Subject: [PATCH 1/2] IB/mlx5: posting klm/mtt list inline in the send queue
 for reg_wr
In-Reply-To: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
Message-ID: <1509295101-14081-2-git-send-email-idanb@mellanox.com>

From: Idan Burstein <idanb at mellanox.com>

As most kernel RDMA ULPs, NVMe over Fabrics in its secure
"register_always" mode registers and invalidates user buffer
upon each IO.

Today the mlx5 driver is posting the registration work
request using scatter/gather entry for the MTT/KLM list.

The fetch of the MTT/KLM list becomes the bottleneck in
number of IO operation could be done by NVMe over Fabrics
host driver on a single adapter as shown below.

This patch is adding the support for inline registration
work request upon MTT/KLM list of size <=64B.

The result for NVMe over Fabrics is increase of x3.6 for small
IOs as shown below, I expect other ULPs (e.g iSER, NFS over RDMA)
performance to be enhanced as well.

The following results were taken against a single nvme over fabrics
subsystem with a single namespace backed by null_blk:

Block Size       s/g reg_wr            inline reg_wr
++++++++++     +++++++++++++++        ++++++++++++++++
512B            1446.6K/8.57%          5224.2K/76.21%
1KB             1390.6K/8.5%           4656.7K/71.69%
2KB             1343.8K/8.6%           3410.3K/38.96%
4KB             1254.8K/8.39%          2033.6K/15.86%
8KB             1079.5K/7.7%           1143.1K/7.27%
16KB            603.4K/3.64%           593.8K/3.4%
32KB            294.8K/2.04%           293.7K/1.98%
64KB            138.2K/1.32%           141.6K/1.26%

Reviewed-by: Max Gurtovoy <maxg at mellanox.com>
Signed-off-by: Idan Burstein <idanb at mellanox.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 41 +++++++++++++++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index acb79d3..b25b93d 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -53,6 +53,7 @@ enum {
 
 enum {
 	MLX5_IB_SQ_STRIDE	= 6,
+	MLX5_IB_SQ_UMR_INLINE_THRESHOLD = 64,
 };
 
 static const u32 mlx5_ib_opcode[] = {
@@ -295,7 +296,8 @@ static int sq_overhead(struct ib_qp_init_attr *attr)
 			max(sizeof(struct mlx5_wqe_atomic_seg) +
 			    sizeof(struct mlx5_wqe_raddr_seg),
 			    sizeof(struct mlx5_wqe_umr_ctrl_seg) +
-			    sizeof(struct mlx5_mkey_seg));
+			    sizeof(struct mlx5_mkey_seg) +
+			    MLX5_IB_SQ_UMR_INLINE_THRESHOLD / MLX5_IB_UMR_OCTOWORD);
 		break;
 
 	case IB_QPT_XRC_TGT:
@@ -3164,13 +3166,15 @@ static __be64 sig_mkey_mask(void)
 }
 
 static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
-			    struct mlx5_ib_mr *mr)
+			    struct mlx5_ib_mr *mr, bool umr_inline)
 {
 	int size = mr->ndescs * mr->desc_size;
 
 	memset(umr, 0, sizeof(*umr));
 
 	umr->flags = MLX5_UMR_CHECK_NOT_FREE;
+	if (umr_inline)
+		umr->flags |= MLX5_UMR_INLINE;
 	umr->xlt_octowords = cpu_to_be16(get_xlt_octo(size));
 	umr->mkey_mask = frwr_mkey_mask();
 }
@@ -3341,6 +3345,24 @@ static void set_reg_data_seg(struct mlx5_wqe_data_seg *dseg,
 	dseg->lkey = cpu_to_be32(pd->ibpd.local_dma_lkey);
 }
 
+static void set_reg_umr_inline_seg(void *seg, struct mlx5_ib_qp *qp,
+				   struct mlx5_ib_mr *mr, int mr_list_size)
+{
+	void *qend = qp->sq.qend;
+	void *addr = mr->descs;
+	int copy;
+
+        if (unlikely(seg + mr_list_size > qend)) {
+                copy = qend - seg;
+                memcpy(seg, addr, copy);
+                addr += copy;
+                mr_list_size -= copy;
+                seg = mlx5_get_send_wqe(qp, 0);
+        }
+        memcpy(seg, addr, mr_list_size);
+        seg += mr_list_size;
+}
+
 static __be32 send_ieth(struct ib_send_wr *wr)
 {
 	switch (wr->opcode) {
@@ -3735,6 +3757,8 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 {
 	struct mlx5_ib_mr *mr = to_mmr(wr->mr);
 	struct mlx5_ib_pd *pd = to_mpd(qp->ibqp.pd);
+	int mr_list_size = mr->ndescs * mr->desc_size;
+	bool umr_inline = mr_list_size <= MLX5_IB_SQ_UMR_INLINE_THRESHOLD;
 
 	if (unlikely(wr->wr.send_flags & IB_SEND_INLINE)) {
 		mlx5_ib_warn(to_mdev(qp->ibqp.device),
@@ -3742,7 +3766,7 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 		return -EINVAL;
 	}
 
-	set_reg_umr_seg(*seg, mr);
+	set_reg_umr_seg(*seg, mr, umr_inline);
 	*seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
 	*size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
 	if (unlikely((*seg == qp->sq.qend)))
@@ -3754,9 +3778,14 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 	if (unlikely((*seg == qp->sq.qend)))
 		*seg = mlx5_get_send_wqe(qp, 0);
 
-	set_reg_data_seg(*seg, mr, pd);
-	*seg += sizeof(struct mlx5_wqe_data_seg);
-	*size += (sizeof(struct mlx5_wqe_data_seg) / 16);
+	if (umr_inline) {
+		set_reg_umr_inline_seg(*seg, qp, mr, mr_list_size);
+		*size += get_xlt_octo(mr_list_size);
+	} else {
+                set_reg_data_seg(*seg, mr, pd);
+                *seg += sizeof(struct mlx5_wqe_data_seg);
+                *size += (sizeof(struct mlx5_wqe_data_seg) / 16);
+	}
 
 	return 0;
 }
-- 
1.8.3.1



From sagi at grimberg.me  Sun Oct 29 09:59:17 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 18:59:17 +0200
Subject: [PATCH 0/2] Performance Improvents for Secured Mode NVMe over
 Fabrics and other RDMA ULPs
In-Reply-To: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
Message-ID: <e1ba4993-5c77-4754-967c-4da8ec34bf77@grimberg.me>

Hi Idan,

One minor nit, there is no "secured mode", its the normal sane
mode. The other mode compromises insecurity for performance.

> The following two patches are including performance
> optimizations for RDMA kernel ULPs that register/invalidate
> per IO (e.g. NVMe over Fabrics, iSER, NFS RDMA). The patches
> improve the IOPs could be achieved by a single network
> adapter with the above ULPs, specifically for NVMe over
> Fabrics the IOPs have improved x5 for small IO reads could
> be achieved in Linux using a single network adapter.

That is very good! Thanks for doing this!
You should note that its mlx5 specific enhancement, which is
an important device, but its not a performance improvement for
RDMA ULPs as it is.

What about mlx4 btw?


From sagi at grimberg.me  Sun Oct 29 10:09:24 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Sun, 29 Oct 2017 19:09:24 +0200
Subject: [PATCH 1/2] IB/mlx5: posting klm/mtt list inline in the send
 queue for reg_wr
In-Reply-To: <1509295101-14081-2-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-2-git-send-email-idanb@mellanox.com>
Message-ID: <a97dd33b-1185-b2db-6df2-a636be963c71@grimberg.me>


> As most kernel RDMA ULPs, NVMe over Fabrics in its secure
> "register_always" mode registers and invalidates user buffer
> upon each IO.

Again, lets drop the secured mode language.

> diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
> index acb79d3..b25b93d 100644
> --- a/drivers/infiniband/hw/mlx5/qp.c
> +++ b/drivers/infiniband/hw/mlx5/qp.c
> @@ -53,6 +53,7 @@ enum {
>   
>   enum {
>   	MLX5_IB_SQ_STRIDE	= 6,
> +	MLX5_IB_SQ_UMR_INLINE_THRESHOLD = 64,
>   };

Is this a device capability? Is it true for all mlx5 devices?
can we get it from the FW instead?

Otherwise, looks good to me,

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From maxg at mellanox.com  Sun Oct 29 10:09:41 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Sun, 29 Oct 2017 19:09:41 +0200
Subject: [PATCH 0/2] Performance Improvents for Secured Mode NVMe over
 Fabrics and other RDMA ULPs
In-Reply-To: <e1ba4993-5c77-4754-967c-4da8ec34bf77@grimberg.me>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <e1ba4993-5c77-4754-967c-4da8ec34bf77@grimberg.me>
Message-ID: <8cb59a0a-c902-0c0e-27a8-fdf9b98982ac@mellanox.com>



On 10/29/2017 6:59 PM, Sagi Grimberg wrote:
> Hi Idan,
> 
> One minor nit, there is no "secured mode", its the normal sane
> mode. The other mode compromises insecurity for performance.
> 
>> The following two patches are including performance
>> optimizations for RDMA kernel ULPs that register/invalidate
>> per IO (e.g. NVMe over Fabrics, iSER, NFS RDMA). The patches
>> improve the IOPs could be achieved by a single network
>> adapter with the above ULPs, specifically for NVMe over
>> Fabrics the IOPs have improved x5 for small IO reads could
>> be achieved in Linux using a single network adapter.
> 
> That is very good! Thanks for doing this!
> You should note that its mlx5 specific enhancement, which is
> an important device, but its not a performance improvement for
> RDMA ULPs as it is.

Sagi,

the subject refers to both patches :)
And it mentions that more improvements are in the pipe, that will 
improve the small write IOs for NVMEoF (at first stage).

> 
> What about mlx4 btw?

We still haven't try to improve the mlx4, but it's on our plate.




From leon at kernel.org  Sun Oct 29 10:43:21 2017
From: leon at kernel.org (Leon Romanovsky)
Date: Sun, 29 Oct 2017 19:43:21 +0200
Subject: [PATCH 0/2] Performance Improvents for Secured Mode NVMe over
 Fabrics and other RDMA ULPs
In-Reply-To: <8cb59a0a-c902-0c0e-27a8-fdf9b98982ac@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <e1ba4993-5c77-4754-967c-4da8ec34bf77@grimberg.me>
 <8cb59a0a-c902-0c0e-27a8-fdf9b98982ac@mellanox.com>
Message-ID: <20171029174321.GM16127@mtr-leonro.local>

On Sun, Oct 29, 2017 at 07:09:41PM +0200, Max Gurtovoy wrote:
> > What about mlx4 btw?
>
> We still haven't try to improve the mlx4, but it's on our plate.

Max and Idan thanks for doing it.

The improvements look good, but I wonder what type of testing did you do?
Did you run full verification suite on it?

Thanks

>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20171029/af83295b/attachment.sig>

From jgg at ziepe.ca  Sun Oct 29 10:52:37 2017
From: jgg at ziepe.ca (Jason Gunthorpe)
Date: Sun, 29 Oct 2017 11:52:37 -0600
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <1509295101-14081-3-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
Message-ID: <20171029175237.GD4488@ziepe.ca>

On Sun, Oct 29, 2017 at 06:38:21PM +0200, idanb at mellanox.com wrote:
  
> +static bool remote_invalidation = true;
> +module_param(remote_invalidation, bool, 0444);
> +MODULE_PARM_DESC(remote_invalidation,
> +	 "request remote invalidation from subsystem (default: true)");

Please no module options.

If your device has a performance anomaly that makes
SEND_WITH_INVALIDATE slower than local invalidation, then we need to
talk about having a new verbs flag for this situation so all the ULPs
can avoid using SEND_WITH_INVALIDATE..

Sagi, is this really an apples to apples comparison?

When I was talking with Chuck on NFS there were some various issues on
that side that made the local invalidate run faster, but it wasn't
actually working properly.

Does nvme wait for the local invalidate to finish before moving on?

Jason


From chuck.lever at oracle.com  Sun Oct 29 11:24:23 2017
From: chuck.lever at oracle.com (Chuck Lever)
Date: Sun, 29 Oct 2017 14:24:23 -0400
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <1509295101-14081-3-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
Message-ID: <87A0B150-CE67-4C8C-914E-53F66411E1BB@oracle.com>


> On Oct 29, 2017, at 12:38 PM, idanb at mellanox.com wrote:
> 
> From: Idan Burstein <idanb at mellanox.com>
> 
> NVMe over Fabrics in its secure "register_always" mode
> registers and invalidates the user buffer upon each IO.
> The protocol enables the host to request the susbsystem
> to use SEND WITH INVALIDATE operation while returning the
> response capsule and invalidate the local key
> (remote_invalidation).
> In some HW implementations, the local network adapter may
> perform better while using local invalidation operations.
> 
> The results below show that running with local invalidation
> rather then remote invalidation improve the iops one could
> achieve by using the ConnectX-5Ex network adapter by x1.36 factor.
> Nevertheless, using local invalidation induce more CPU overhead
> than enabling the target to invalidate remotly, therefore,
> because there is a CPU% vs IOPs limit tradeoff we propose to
> have module parameter to define whether to request remote
> invalidation or not.
> 
> The following results were taken against a single nvme over fabrics
> subsystem with a single namespace backed by null_blk:
> 
> Block Size       s/g reg_wr      inline reg_wr    inline reg_wr + local inv
> ++++++++++++   ++++++++++++++   ++++++++++++++++ +++++++++++++++++++++++++++
> 512B            1446.6K/8.57%    5224.2K/76.21%   7143.3K/79.72%
> 1KB             1390.6K/8.5%     4656.7K/71.69%   5860.6K/55.45%
> 2KB             1343.8K/8.6%     3410.3K/38.96%   4106.7K/55.82%
> 4KB             1254.8K/8.39%    2033.6K/15.86%   2165.3K/17.48%
> 8KB             1079.5K/7.7%     1143.1K/7.27%    1158.2K/7.33%
> 16KB            603.4K/3.64%     593.8K/3.4%      588.9K/3.77%
> 32KB            294.8K/2.04%     293.7K/1.98%     294.4K/2.93%
> 64KB            138.2K/1.32%     141.6K/1.26%     135.6K/1.34%

Units reported here are KIOPS and %CPU ? What was the benchmark?

Was any root cause analysis done to understand why the IOPS
rate changes without RI? Reduction in avg RTT? Fewer long-
running outliers? Lock contention in the ULP?

I am curious enough to add a similar setting to NFS/RDMA,
now that I have mlx5 devices.


> Signed-off-by: Max Gurtovoy <maxg at mellanox.com>
> Signed-off-by: Idan Burstein <idanb at mellanox.com>
> ---
> drivers/nvme/host/rdma.c | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 92a03ff..7f8225d 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -146,6 +146,11 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl)
> MODULE_PARM_DESC(register_always,
> 	 "Use memory registration even for contiguous memory regions");
> 
> +static bool remote_invalidation = true;
> +module_param(remote_invalidation, bool, 0444);
> +MODULE_PARM_DESC(remote_invalidation,
> +	 "request remote invalidation from subsystem (default: true)");

The use of a module parameter would be awkward in systems
that have a heterogenous collection of HCAs.


> +
> static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
> 		struct rdma_cm_event *event);
> static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
> @@ -1152,8 +1157,9 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
> 	sg->addr = cpu_to_le64(req->mr->iova);
> 	put_unaligned_le24(req->mr->length, sg->length);
> 	put_unaligned_le32(req->mr->rkey, sg->key);
> -	sg->type = (NVME_KEY_SGL_FMT_DATA_DESC << 4) |
> -			NVME_SGL_FMT_INVALIDATE;
> +	sg->type = NVME_KEY_SGL_FMT_DATA_DESC << 4;
> +	if (remote_invalidation)
> +		sg->type |= NVME_SGL_FMT_INVALIDATE;
> 
> 	return 0;
> }
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever





From guanjunxiong at huawei.com  Sun Oct 29 20:37:55 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Mon, 30 Oct 2017 11:37:55 +0800
Subject: [PATCH 16/17] nvme: implement multipath access to nvme subsystems
In-Reply-To: <20171023145126.2471-17-hch@lst.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-17-hch@lst.de>
Message-ID: <70b681c5-5fae-ae40-2ff6-60cf75d52616@huawei.com>



On 2017/10/23 22:51, Christoph Hellwig wrote:
> @@ -2427,20 +2681,46 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
>  	if (ret) {
>  		dev_err(ctrl->device,
>  			"duplicate IDs for nsid %d\n", nsid);
> -		goto out_free_head;
> +		goto out_release_instance;
>  	}
>  
> +	ret = -ENOMEM;
> +	q = blk_alloc_queue_node(GFP_KERNEL, NUMA_NO_NODE);
> +	if (!q)
> +		goto out_free_head;
> +	q->queuedata = head;
> +	blk_queue_make_request(q, nvme_ns_head_make_request);
> +	q->poll_fn = nvme_ns_head_poll;
> +	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
> +	/* set to a default value for 512 until disk is validated */
> +	blk_queue_logical_block_size(q, 512);
> +	nvme_set_queue_limits(ctrl, q);
> +
> +	head->disk = alloc_disk(0);
> +	if (!head->disk)
> +		goto out_cleanup_queue;
> +	head->disk->fops = &nvme_ns_head_ops;
> +	head->disk->private_data = head;
> +	head->disk->queue = q;
> +	head->disk->flags = GENHD_FL_EXT_DEVT;
> +	sprintf(head->disk->disk_name, "nvme%dn%d",
> +			ctrl->subsys->instance, nsid);

Is it okay to use head->instance instead of nsid for disk name nvme#n# ?
Becuase _nsid_ sets are not continuous sometimes, so disk name is ugly in that case.




From guanjunxiong at huawei.com  Sun Oct 29 21:09:39 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Mon, 30 Oct 2017 12:09:39 +0800
Subject: [PATCH 06/17] block: introduce GENHD_FL_HIDDEN
In-Reply-To: <ae4180e2-4c1b-a5f0-5c26-7cec692e6d50@suse.de>
References: <20171023145126.2471-1-hch@lst.de>
 <20171023145126.2471-7-hch@lst.de> <20171024213959.GB23307@redhat.com>
 <20171028063825.GA14261@lst.de> <20171028141745.GA32039@redhat.com>
 <ae4180e2-4c1b-a5f0-5c26-7cec692e6d50@suse.de>
Message-ID: <7cabd9cf-61e7-d53e-d0d5-6af90f2a2ce8@huawei.com>

Hi Christoph, Mike and Hannes

On 2017/10/29 18:01, Hannes Reinecke wrote:
> After all, Linux is about choice, not about forcing users to do things
> in one way only.


I have added an option CONFIG_NVME_SHOW_CTRL_BLK_DEV to show per-controller
block devices nodes.
It is tested simply that dm-multipath and nvme-mpath can coexist.

This patch, based on the Christoph V5 of nvme-mpath,  is in the following.

Does it look good for you ?

Regards
Guan

--
>From de3f446af6591d68ef84333138e744f12db4d695 Mon Sep 17 00:00:00 2001
From: Junxiong Guan <guanjunxiong at huawei.com>
Date: Mon, 30 Oct 2017 04:59:20 -0400
Subject: [PATCH] nvme: add an option to show per-controller block devices
 nodes

Signed-off-by: Junxiong Guan <guanjunxiong at huawei.com>
---
 drivers/nvme/host/Kconfig |  9 ++++++
 drivers/nvme/host/core.c  | 80 +++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 87 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 46d6cb1e03bd..725bff035f38 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -13,6 +13,15 @@ config BLK_DEV_NVME
 	  To compile this driver as a module, choose M here: the
 	  module will be called nvme.

+config NVME_SHOW_CTRL_BLK_DEV
+	bool "Show per-controller block devices of NVMe"
+	depends on NVME_CORE
+	---help---
+	  This adds support to show per-controller block devices nodes in
+	  the user space.
+
+	  If unsure, say N.
+
 config NVME_FABRICS
 	tristate

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 334735db90c8..ae37e274108c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -72,6 +72,9 @@ static DEFINE_IDA(nvme_subsystems_ida);
 static LIST_HEAD(nvme_subsystems);
 static DEFINE_MUTEX(nvme_subsystems_lock);

+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+static DEFINE_SPINLOCK(dev_list_lock);
+#endif
 static DEFINE_IDA(nvme_instance_ida);
 static dev_t nvme_chr_devt;
 static struct class *nvme_class;
@@ -357,6 +360,14 @@ static void nvme_free_ns(struct kref *kref)

 	if (ns->ndev)
 		nvme_nvm_unregister(ns);
+
+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+	if (ns->disk) {
+		spin_lock(&dev_list_lock);
+		ns->disk->private_data = NULL;
+		spin_unlock(&dev_list_lock);
+	}
+#endif
 	put_disk(ns->disk);
 	nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
@@ -1127,10 +1138,32 @@ static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd,
 	}
 }

-/* should never be called due to GENHD_FL_HIDDEN */
 static int nvme_open(struct block_device *bdev, fmode_t mode)
 {
+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+	struct nvme_ns *ns;
+
+	spin_lock(&dev_list_lock);
+	ns = bdev->bd_disk->private_data;
+	if (ns) {
+		if (!kref_get_unless_zero(&ns->kref))
+			goto fail;
+		if (!try_module_get(ns->ctrl->ops->module))
+			goto fail_put_ns;
+	}
+	spin_unlock(&dev_list_lock);
+
+	return ns ? 0 : -ENXIO;
+
+fail_put_ns:
+	kref_put(&ns->kref, nvme_free_ns);
+fail:
+	spin_unlock(&dev_list_lock);
+	return -ENXIO;
+#else
+	/* should never be called due to GENHD_FL_HIDDEN */
 	return WARN_ON_ONCE(-ENXIO);
+#endif /* CONFIG_NVME_SHOW_CTRL_BLK_DEV */
 }

 static int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo)
@@ -1392,10 +1425,15 @@ static char nvme_pr_type(enum pr_type type)
 static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 				u64 key, u64 sa_key, u8 op)
 {
+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+	struct nvme_ns *ns = bdev->bd_disk->private_data;
+	struct nvme_ns_head *head = ns->head;
+#else
 	struct nvme_ns_head *head = bdev->bd_disk->private_data;
 	struct nvme_ns *ns;
-	struct nvme_command c;
 	int srcu_idx, ret;
+#endif
+	struct nvme_command c;
 	u8 data[16] = { 0, };

 	put_unaligned_le64(key, &data[0]);
@@ -1406,6 +1444,9 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 	c.common.nsid = cpu_to_le32(head->ns_id);
 	c.common.cdw10[0] = cpu_to_le32(cdw10);

+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+	return nvme_submit_sync_cmd(ns->queue, &c, data, 16);
+#else
 	srcu_idx = srcu_read_lock(&head->srcu);
 	ns = nvme_find_path(head);
 	if (likely(ns))
@@ -1413,7 +1454,9 @@ static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 	else
 		ret = -EWOULDBLOCK;
 	srcu_read_unlock(&head->srcu, srcu_idx);
+
 	return ret;
+#endif
 }

 static int nvme_pr_register(struct block_device *bdev, u64 old,
@@ -1492,6 +1535,34 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 EXPORT_SYMBOL_GPL(nvme_sec_submit);
 #endif /* CONFIG_BLK_SED_OPAL */

+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+static void nvme_release(struct gendisk *disk, fmode_t mode)
+{
+	struct nvme_ns *ns = disk->private_data;
+
+	module_put(ns->ctrl->ops->module);
+	nvme_put_ns(ns);
+}
+
+static int nvme_ioctl(struct block_device *bdev, fmode_t mode,
+		unsigned int cmd, unsigned long arg)
+{
+	struct nvme_ns *ns = bdev->bd_disk->private_data;
+
+	return nvme_ns_ioctl(ns, cmd, arg);
+}
+
+static const struct block_device_operations nvme_fops = {
+	.owner		= THIS_MODULE,
+	.ioctl		= nvme_ioctl,
+	.compat_ioctl	= nvme_ioctl,
+	.open		= nvme_open,
+	.release	= nvme_release,
+	.getgeo		= nvme_getgeo,
+	.revalidate_disk= nvme_revalidate_disk,
+	.pr_ops		=&nvme_pr_ops,
+};
+#else
 /*
  * While we don't expose the per-controller devices to userspace we still
  * need valid file operations for them, for one because the block layer
@@ -1503,6 +1574,7 @@ static const struct block_device_operations nvme_fops = {
 	.open		= nvme_open,
 	.revalidate_disk= nvme_revalidate_disk,
 };
+#endif /* CONFIG_NVME_SHOW_CTRL_BLK_DEV */

 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u64 cap, bool enabled)
 {
@@ -2875,7 +2947,11 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	disk->fops = &nvme_fops;
 	disk->private_data = ns;
 	disk->queue = ns->queue;
+#ifdef CONFIG_NVME_SHOW_CTRL_BLK_DEV
+	disk->flags = GENHD_FL_EXT_DEVT;
+#else
 	disk->flags = GENHD_FL_HIDDEN;
+#endif
 	memcpy(disk->disk_name, disk_name, DISK_NAME_LEN);
 	ns->disk = disk;

-- 
2.11.1




From james.smart at broadcom.com  Sun Oct 29 21:43:14 2017
From: james.smart at broadcom.com (James Smart)
Date: Sun, 29 Oct 2017 21:43:14 -0700
Subject: [PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller
 rescan
In-Reply-To: <286e4dc4-af2c-3482-4345-a7b9984da994@grimberg.me>
References: <20171028172114.2045-1-jsmart2021@gmail.com>
 <286e4dc4-af2c-3482-4345-a7b9984da994@grimberg.me>
Message-ID: <ed37c118-6a9b-b4ab-dc8c-27eaa0c178f9@broadcom.com>

On 10/29/2017 9:11 AM, Sagi Grimberg wrote:
> James,
>
>> A transport may have a transport-specific mechanism that can signal
>> when discovery controller content has changed and request a host
>> to reconnect to the discovery controller.
>>
>> FC is such a transport. RSCNs may be generated by the FC port with
>> the discovery server, with the RSCNs then broadcast to the FC-NVME
>> hosts. A host, upon receiving the RSCN, would validate connectivity
>> then initiate a discovery controller rescan, allowing new subsystems
>> to be connected to or updating subsystem connectivity tables.
>
> How does this fit with possible future discovery enhancements discussed
> in nvme TWG right now? The notification for discovery records change
> will not be transport specific in the future.

Its independent yet can coexist very well.

Currently, the FC-NVME standard strongly suggests a discovery controller 
on each target. Having this simple notification avoids lots of 
long-lived connections and rescans are done only when/where needed - 
with no change in any standard. It works very well on current products.? 
There's nothing preventing it coexisting with long-lived discovery 
controller connections and other configurations of discovery servers.

I don't buy into your last statement yet. Long lived discovery 
connections are fine and an admin can already configure a set of systems 
as they describe. I also expect any attempt to mandate centralized 
discovery controllers to have a fair number of issues to be worked out 
not just with hosts but with targets and cross-fabric issues and I 
believe there will be pushback to give complete control of presentation 
to third party entity.? In some respects, the history of iSNS/SLP for 
iSCSI gave a brief example of trying to go down this path.

-- james



From james.smart at broadcom.com  Sun Oct 29 21:43:46 2017
From: james.smart at broadcom.com (James Smart)
Date: Sun, 29 Oct 2017 21:43:46 -0700
Subject: [PATCH 1/2] nvme_fc: fix localport resume using stale values
In-Reply-To: <4640f307-5d61-838b-62d4-c8310ff4bd3b@grimberg.me>
References: <20171027172805.13495-1-jsmart2021@gmail.com>
 <20171027172805.13495-2-jsmart2021@gmail.com>
 <4640f307-5d61-838b-62d4-c8310ff4bd3b@grimberg.me>
Message-ID: <3d7cff5b-5e23-b9be-33c2-474b6393f184@broadcom.com>

On 10/29/2017 5:28 AM, Sagi Grimberg wrote:
>
>> The localport resume was not updating the lldd ops structure. If the
>> lldd is unloaded and reloaded, the ops pointers will differ.
>
> Question, why is the lport kept around if the lldd unloaded?

It's kept around due to ref counting of the attached rports and 
controllers who have not timed out and torn down.? The fact that the 
lldd could be unloaded is an artifact of supporting the reg/unreg and 
the fact that the transport doesn't care what the driver does after the 
unreg is done, and the re-reg only cares that it's done quickly enough 
that there are still references so the device and it can revive the old 
structures.? The reg/unreg/re-reg support is mainly there to support the 
lldd performing an adapter reset for some type of error recovery (could 
come from the SCSI stack requesting it for error escalation). Thus the 
lldd can reset the adapter, perform it's normal init steps again an 
re-reg the lport and rports, and the controllers resume under their 
ctlr_loss_tmo/dev_loss_tmo timers and consumers of the nvme devices are 
not affected other than a short pause.

-- james



From leon at kernel.org  Sun Oct 29 22:21:10 2017
From: leon at kernel.org (Leon Romanovsky)
Date: Mon, 30 Oct 2017 07:21:10 +0200
Subject: [PATCH 0/2] Performance Improvents for Secured Mode NVMe over
 Fabrics and other RDMA ULPs
In-Reply-To: <AM0PR0502MB38906F57CD68C3707F8FE765C5580@AM0PR0502MB3890.eurprd05.prod.outlook.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <e1ba4993-5c77-4754-967c-4da8ec34bf77@grimberg.me>
 <8cb59a0a-c902-0c0e-27a8-fdf9b98982ac@mellanox.com>
 <20171029174321.GM16127@mtr-leonro.local>
 <AM0PR0502MB38906F57CD68C3707F8FE765C5580@AM0PR0502MB3890.eurprd05.prod.outlook.com>
Message-ID: <20171030052110.GP16127@mtr-leonro.local>

On Sun, Oct 29, 2017 at 06:03:43PM +0000, Idan Burstein wrote:
> Leon,
>
> We only did performance testing at the block level.

Excellent, so please, as a Mellanox employee, run it through our
standard submission channels, so at least we will be sure that you
are not breaking anything.

Please talk with Max, he did it before and he knows how to do it right.

Thanks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20171030/46aec3a2/attachment.sig>

From sagi at grimberg.me  Mon Oct 30 01:11:54 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 30 Oct 2017 10:11:54 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <1509295101-14081-3-git-send-email-idanb@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
Message-ID: <4423f96f-bf42-3603-aa6a-fa259a1d09d1@grimberg.me>

Idan,

> NVMe over Fabrics in its secure "register_always" mode
> registers and invalidates the user buffer upon each IO.
> The protocol enables the host to request the susbsystem
> to use SEND WITH INVALIDATE operation while returning the
> response capsule and invalidate the local key
> (remote_invalidation).
> In some HW implementations, the local network adapter may
> perform better while using local invalidation operations.

This is a device quirk and if you want to optimize for this
you need to expose it via verbs interface. We won't add a module
parameter for implementation specific stuff.

Note that we currently need to rework the case where we use local
invalidates and properly wait for them to complete.


From sagi at grimberg.me  Mon Oct 30 01:14:08 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 30 Oct 2017 10:14:08 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <20171029175237.GD4488@ziepe.ca>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
Message-ID: <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>


> Sagi, is this really an apples to apples comparison?

No

> When I was talking with Chuck on NFS there were some various issues on
> that side that made the local invalidate run faster, but it wasn't
> actually working properly.
> 
> Does nvme wait for the local invalidate to finish before moving on?

Not really, and we should, I'll get to it.


From idanb at mellanox.com  Mon Oct 30 01:38:18 2017
From: idanb at mellanox.com (idanb)
Date: Mon, 30 Oct 2017 10:38:18 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
 <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
Message-ID: <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>



On 10/30/2017 10:14 AM, Sagi Grimberg wrote:
>
>> Sagi, is this really an apples to apples comparison?
>
> No
>
>> When I was talking with Chuck on NFS there were some various issues on
>> that side that made the local invalidate run faster, but it wasn't
>> actually working properly.
>>
>> Does nvme wait for the local invalidate to finish before moving on?
>
> Not really, and we should, I'll get to it.

I agree we should fix this, nevertheless I don't expect this to create 
an IO rate issue for long queue depths.
Indeed we will have a tradeoff here of latency vs rate.



From idanb at mellanox.com  Mon Oct 30 01:45:12 2017
From: idanb at mellanox.com (idanb)
Date: Mon, 30 Oct 2017 10:45:12 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <4423f96f-bf42-3603-aa6a-fa259a1d09d1@grimberg.me>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <4423f96f-bf42-3603-aa6a-fa259a1d09d1@grimberg.me>
Message-ID: <2f9723ae-93f4-aeec-22b0-31634d52b3aa@mellanox.com>



On 10/30/2017 10:11 AM, Sagi Grimberg wrote:
> Idan,
>
>> NVMe over Fabrics in its secure "register_always" mode
>> registers and invalidates the user buffer upon each IO.
>> The protocol enables the host to request the susbsystem
>> to use SEND WITH INVALIDATE operation while returning the
>> response capsule and invalidate the local key
>> (remote_invalidation).
>> In some HW implementations, the local network adapter may
>> perform better while using local invalidation operations.
>
> This is a device quirk and if you want to optimize for this
> you need to expose it via verbs interface. We won't add a module
> parameter for implementation specific stuff.

After our discussion yesterday I was thinking it may be better to hide
it within the nvme host driver while having a messages size threshold for
using send_with_invalidate. To make it device agnostic we will need
to come out with an idea of how to expose such limitation.

>
> Note that we currently need to rework the case where we use local
> invalidates and properly wait for them to complete.



From sagi at grimberg.me  Mon Oct 30 02:44:26 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 30 Oct 2017 11:44:26 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
 <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
 <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>
Message-ID: <38523e67-fa00-dd03-5b6f-34cd6f863c8f@grimberg.me>


> I agree we should fix this, nevertheless I don't expect this to create 
> an IO rate issue for long queue depths.
> Indeed we will have a tradeoff here of latency vs rate.

I suggest we hold back with this one for now, I have fixes for
the local invalidate flow. Once we get them in we can see if
this approach makes a difference, and if so, we need to expose
it through a generic mechanism.


From guanjunxiong at huawei.com  Mon Oct 30 03:14:30 2017
From: guanjunxiong at huawei.com (Guan Junxiong)
Date: Mon, 30 Oct 2017 18:14:30 +0800
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <20171020223746.31287-1-keith.busch@intel.com>
References: <20171020223746.31287-1-keith.busch@intel.com>
Message-ID: <b05dcfac-0d50-4b90-544e-646bbfce57bb@huawei.com>


On 2017/10/21 6:37, Keith Busch wrote:
>  static void nvme_async_event_work(struct work_struct *work)
>  {
>  	struct nvme_ctrl *ctrl =
>  		container_of(work, struct nvme_ctrl, async_event_work);
>  
> +	nvme_aen_uevent(ctrl);
>  	ctrl->ops->submit_async_event(ctrl);
>  }

Is this based on 4.15 or 4.14? I failed to apply it.



From idanb at mellanox.com  Mon Oct 30 03:31:06 2017
From: idanb at mellanox.com (idanb)
Date: Mon, 30 Oct 2017 12:31:06 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <38523e67-fa00-dd03-5b6f-34cd6f863c8f@grimberg.me>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
 <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
 <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>
 <38523e67-fa00-dd03-5b6f-34cd6f863c8f@grimberg.me>
Message-ID: <5783e083-93d4-8e72-c380-03fc54a0291b@mellanox.com>



On 10/30/2017 11:44 AM, Sagi Grimberg wrote:
>
>> I agree we should fix this, nevertheless I don't expect this to 
>> create an IO rate issue for long queue depths.
>> Indeed we will have a tradeoff here of latency vs rate.
>
> I suggest we hold back with this one for now, I have fixes for
> the local invalidate flow. Once we get them in we can see if
> this approach makes a difference, and if so, we need to expose
> it through a generic mechanism.

I suggest that you'll post the patches to the list and we'll investigate
the latency implications, if the implications are negligible, we have a
win here for small IOs.

When will you be able to post your patches?


From sagi at grimberg.me  Mon Oct 30 03:33:18 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 30 Oct 2017 12:33:18 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <5783e083-93d4-8e72-c380-03fc54a0291b@mellanox.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
 <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
 <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>
 <38523e67-fa00-dd03-5b6f-34cd6f863c8f@grimberg.me>
 <5783e083-93d4-8e72-c380-03fc54a0291b@mellanox.com>
Message-ID: <360f892f-b88d-0947-1590-ab1d64d4da13@grimberg.me>



>>> I agree we should fix this, nevertheless I don't expect this to 
>>> create an IO rate issue for long queue depths.
>>> Indeed we will have a tradeoff here of latency vs rate.
>>
>> I suggest we hold back with this one for now, I have fixes for
>> the local invalidate flow. Once we get them in we can see if
>> this approach makes a difference, and if so, we need to expose
>> it through a generic mechanism.
> 
> I suggest that you'll post the patches to the list and we'll investigate
> the latency implications, if the implications are negligible, we have a
> win here for small IOs.
> 
> When will you be able to post your patches?

I can post them now, but I prefer to test them first ;)


From idanb at mellanox.com  Mon Oct 30 05:35:42 2017
From: idanb at mellanox.com (idanb)
Date: Mon, 30 Oct 2017 14:35:42 +0200
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <360f892f-b88d-0947-1590-ab1d64d4da13@grimberg.me>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <20171029175237.GD4488@ziepe.ca>
 <740c93f4-164e-d4e3-97b1-313a0420ae81@grimberg.me>
 <7e038d80-de95-8fb7-e313-825e40c03e88@mellanox.com>
 <38523e67-fa00-dd03-5b6f-34cd6f863c8f@grimberg.me>
 <5783e083-93d4-8e72-c380-03fc54a0291b@mellanox.com>
 <360f892f-b88d-0947-1590-ab1d64d4da13@grimberg.me>
Message-ID: <22964904-2537-51c3-bf39-5b2197aec5b9@mellanox.com>


On 10/30/2017 12:33 PM, Sagi Grimberg wrote:
>
>
>>>> I agree we should fix this, nevertheless I don't expect this to 
>>>> create an IO rate issue for long queue depths.
>>>> Indeed we will have a tradeoff here of latency vs rate.
>>>
>>> I suggest we hold back with this one for now, I have fixes for
>>> the local invalidate flow. Once we get them in we can see if
>>> this approach makes a difference, and if so, we need to expose
>>> it through a generic mechanism.
>>
>> I suggest that you'll post the patches to the list and we'll investigate
>> the latency implications, if the implications are negligible, we have a
>> win here for small IOs.
>>
>> When will you be able to post your patches?
>
> I can post them now, but I prefer to test them first ;)

Sounds good, we will check once you'll post them.
I expect no impact on IOPs,  just latency.



From axboe at kernel.dk  Mon Oct 30 07:23:34 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Mon, 30 Oct 2017 08:23:34 -0600
Subject: [PATCH urgent-4.14] nvme: Fix setting logical block format when
 revalidating
In-Reply-To: <20171027195122.1810-1-keith.busch@intel.com>
References: <20171027195122.1810-1-keith.busch@intel.com>
Message-ID: <fb7b1486-ab1c-05f8-3c3b-7dbaef6d75c3@kernel.dk>

On 10/27/2017 01:51 PM, Keith Busch wrote:
> Revalidating the disk needs to set the logical block format and capacity,
> otherwise it can't figure out if the users modified anything about
> the namespace.

Applied, thanks Keith.

-- 
Jens Axboe



From axboe at kernel.dk  Mon Oct 30 07:29:31 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Mon, 30 Oct 2017 08:29:31 -0600
Subject: [PATCH] nvme: freeze IO accesses around format
In-Reply-To: <20171027230756.GA17245@localhost.localdomain>
References: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
 <20171027230756.GA17245@localhost.localdomain>
Message-ID: <459d7b58-c5b6-44a9-280e-ce8661673b18@kernel.dk>

On 10/27/2017 05:07 PM, Keith Busch wrote:
> On Fri, Oct 27, 2017 at 10:35:58AM -0600, Jens Axboe wrote:
>> @@ -1013,10 +1013,23 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>>  	if (cmd.timeout_ms)
>>  		timeout = msecs_to_jiffies(cmd.timeout_ms);
>>  
>> +	/*
>> +	 * Freeze current access to the device, and prevent new ones, around
>> +	 * a format operation.
>> +	 */
>> +	if (cmd.opcode == nvme_admin_format_nvm) {
>> +		nvme_start_freeze(ctrl);
>> +		nvme_wait_freeze(ctrl);
>> +	}
>> +
>>  	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
>>  			(void __user *)(uintptr_t)cmd.addr, cmd.data_len,
>>  			(void __user *)(uintptr_t)cmd.metadata, cmd.metadata,
>>  			0, &cmd.result, timeout);
>> +
>> +	if (cmd.opcode == nvme_admin_format_nvm)
>> +		nvme_unfreeze(ctrl);
>> +
> 
> Testing some obscure vendor specific IO commands, I noticed this
> will deadlock if an IO command opcode happens to be the same as an
> nvme_admin_format_nvm command since the IO queues are all frozen.
> 
> You can check 'ns == NULL' before checking the opcode.

Yeah good point, it should have checked for this being the admin
queue target.

> I also have this patch that's more generic and will perform actions
> based on the controller's command effects:

Looks fine to me, one minor comment below.


> +static void nvme_get_admin_effects(struct nvme_ctrl *ctrl, u8 opcode, u32 *effects)
> +{
> +	if (ctrl->effects)
> +		*effects = le32_to_cpu(ctrl->effects->acs[opcode]);
> +
> +	if (*effects != 0)
> +		return;
> +
> +	/*
> +	 * Set known effects for opcodes if the controller doesn't support
> +	 * reporting the command's effects.
> +	 */
> +	switch (opcode) {
> +	case nvme_admin_format_nvm:
> +		*effects = NVME_CMD_FX_CSUPP | NVME_CMD_FX_LBCC |
> +					NVME_CMD_FX_CSE_MASK;
> +		break;
> +	default:
> +		break;
> +	}
> +}

Returning the value in a passed in pointer is pretty ugly, I only
generally think it's acceptable if you have to pass back some error as
well. And it's still nasty in that case. How about just returning an u32
from this function?


-- 
Jens Axboe



From axboe at kernel.dk  Mon Oct 30 07:30:04 2017
From: axboe at kernel.dk (Jens Axboe)
Date: Mon, 30 Oct 2017 08:30:04 -0600
Subject: [PATCH] nvme: freeze IO accesses around format
In-Reply-To: <20171028064508.GA20977@infradead.org>
References: <6951d74c-e765-1b5d-6e39-d88d261bf9b9@kernel.dk>
 <20171028064508.GA20977@infradead.org>
Message-ID: <ac29e201-7d60-d698-d6a3-9b4c0307c91f@kernel.dk>

On 10/28/2017 12:45 AM, Christoph Hellwig wrote:
> And what about sanitize?  I guess instead of looking at opcodes directly
> we'd better look at the indications in the Commands Supported and Effects
> log.  That's what the Windows driver does, btw.

Isn't that exactly what Keith's patch did?

-- 
Jens Axboe



From keith.busch at intel.com  Mon Oct 30 08:18:01 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 30 Oct 2017 09:18:01 -0600
Subject: [PATCHv2 5/5] nvme: Send uevent for unhandled AEN completions
In-Reply-To: <b05dcfac-0d50-4b90-544e-646bbfce57bb@huawei.com>
References: <20171020223746.31287-1-keith.busch@intel.com>
 <b05dcfac-0d50-4b90-544e-646bbfce57bb@huawei.com>
Message-ID: <20171030151800.GC17245@localhost.localdomain>

On Mon, Oct 30, 2017 at 06:14:30PM +0800, Guan Junxiong wrote:
> 
> On 2017/10/21 6:37, Keith Busch wrote:
> >  static void nvme_async_event_work(struct work_struct *work)
> >  {
> >  	struct nvme_ctrl *ctrl =
> >  		container_of(work, struct nvme_ctrl, async_event_work);
> >  
> > +	nvme_aen_uevent(ctrl);
> >  	ctrl->ops->submit_async_event(ctrl);
> >  }
> 
> Is this based on 4.15 or 4.14? I failed to apply it.
> 

This was based off this branch:

  http://git.infradead.org/nvme.git/shortlog/refs/heads/nvme-4.15

starting from commit 134aedc9c ("nvme-fc: correct io timeout behavior")


From chuck.lever at oracle.com  Mon Oct 30 11:18:42 2017
From: chuck.lever at oracle.com (Chuck Lever)
Date: Mon, 30 Oct 2017 14:18:42 -0400
Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter
In-Reply-To: <87A0B150-CE67-4C8C-914E-53F66411E1BB@oracle.com>
References: <1509295101-14081-1-git-send-email-idanb@mellanox.com>
 <1509295101-14081-3-git-send-email-idanb@mellanox.com>
 <87A0B150-CE67-4C8C-914E-53F66411E1BB@oracle.com>
Message-ID: <2FB68C09-F811-44C9-9F48-826C074DDE14@oracle.com>


> On Oct 29, 2017, at 2:24 PM, Chuck Lever <chuck.lever at oracle.com> wrote:
> 
> 
>> On Oct 29, 2017, at 12:38 PM, idanb at mellanox.com wrote:
>> 
>> From: Idan Burstein <idanb at mellanox.com>
>> 
>> NVMe over Fabrics in its secure "register_always" mode
>> registers and invalidates the user buffer upon each IO.
>> The protocol enables the host to request the susbsystem
>> to use SEND WITH INVALIDATE operation while returning the
>> response capsule and invalidate the local key
>> (remote_invalidation).
>> In some HW implementations, the local network adapter may
>> perform better while using local invalidation operations.
>> 
>> The results below show that running with local invalidation
>> rather then remote invalidation improve the iops one could
>> achieve by using the ConnectX-5Ex network adapter by x1.36 factor.
>> Nevertheless, using local invalidation induce more CPU overhead
>> than enabling the target to invalidate remotly, therefore,
>> because there is a CPU% vs IOPs limit tradeoff we propose to
>> have module parameter to define whether to request remote
>> invalidation or not.
>> 
>> The following results were taken against a single nvme over fabrics
>> subsystem with a single namespace backed by null_blk:
>> 
>> Block Size       s/g reg_wr      inline reg_wr    inline reg_wr + local inv
>> ++++++++++++   ++++++++++++++   ++++++++++++++++ +++++++++++++++++++++++++++
>> 512B            1446.6K/8.57%    5224.2K/76.21%   7143.3K/79.72%
>> 1KB             1390.6K/8.5%     4656.7K/71.69%   5860.6K/55.45%
>> 2KB             1343.8K/8.6%     3410.3K/38.96%   4106.7K/55.82%
>> 4KB             1254.8K/8.39%    2033.6K/15.86%   2165.3K/17.48%
>> 8KB             1079.5K/7.7%     1143.1K/7.27%    1158.2K/7.33%
>> 16KB            603.4K/3.64%     593.8K/3.4%      588.9K/3.77%
>> 32KB            294.8K/2.04%     293.7K/1.98%     294.4K/2.93%
>> 64KB            138.2K/1.32%     141.6K/1.26%     135.6K/1.34%
> 
> Units reported here are KIOPS and %CPU ? What was the benchmark?
> 
> Was any root cause analysis done to understand why the IOPS
> rate changes without RI? Reduction in avg RTT? Fewer long-
> running outliers? Lock contention in the ULP?
> 
> I am curious enough to add a similar setting to NFS/RDMA,
> now that I have mlx5 devices.

I see the plan is to change the NVMeoF initiator to wait for
invalidation to complete, and then test again. However, I am
still curious what we're dealing with for NFS/RDMA (which
always waits for invalidation before allowing an RPC to
complete).

I tested here with a pair of EDR CX-4s in RoCE mode. I found
that there is a 15-20us latency penalty when Remote Invalid-
ation is used with RDMA Read, but when used with RDMA Write,
Remote Invalidation behaves as desired. The RDMA Read penalty
vanishes after payloads are larger than about 8kB.

Simple QD=1 iozone test with direct I/O on NFSv3, and a tmpfs
share on the NFS server. In this test, memory registration is
used for all data payloads.

Remote Invalidation enabled, reclen in kB, output in
microseconds:

              kB  reclen    write  rewrite    read    reread
          131072       1       47       54       27       27
          131072       2       61       62       27       28
          131072       4       63       62       28       28
          131072       8       59       65       29       29
          131072      16       67       66       31       32
          131072      32       75       73       42       42
          131072      64       92       87       64       44

Remote Invalidation disabled, reclen in kB, output in
microseconds

              kB  reclen    write  rewrite    read    reread
          131072       1       43       43       32       32
          131072       2       45       52       32       32
          131072       4       48       45       32       32
          131072       8       68       64       33       33
          131072      16       74       60       35       35
          131072      32       85       82       49       41
          131072      64      102       98       63       52

I would expect a similar ~5us latency benefit for both RDMA
Reads and RDMA Writes when Remote Invalidation is in use.
Small I/Os involving RDMA Read are precisely where NFS/RDMA
gains the most benefit from Remote Invalidation, so this
result is disappointing.

Unfortunately, the current version of RPC-over-RDMA does not
have the ability to convey an Rkey for the target to invalid-
ate remotely, though that is one of the features being
considered for the next version.

IOPS results (fio 70/30, multi-thread iozone, etc) are very
clearly worse without Remote Invalidation, so I currently
don't plan to submit a patch that allows RI to be disabled
for some NFS/RDMA mounts. Currently the Linux NFS client
indicates to servers that RI is supported whenever FRWR is
supported on the client's HCA. The server is then free to
decide whether to use Send With Invalidate with any Rkey
in the current RPC.


--
Chuck Lever





From keith.busch at intel.com  Mon Oct 30 12:36:07 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 30 Oct 2017 13:36:07 -0600
Subject: [PATCH] nvme-cli: make command description const
In-Reply-To: <1509215193-13119-1-git-send-email-minwoo.im.dev@gmail.com>
References: <1509215193-13119-1-git-send-email-minwoo.im.dev@gmail.com>
Message-ID: <20171030193606.GG17245@localhost.localdomain>

On Sun, Oct 29, 2017 at 03:26:33AM +0900, Minwoo Im wrote:
> Make desc in sanitize_log() const.
> 
> Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>

Applied, thanks.


From keith.busch at intel.com  Mon Oct 30 12:36:16 2017
From: keith.busch at intel.com (Keith Busch)
Date: Mon, 30 Oct 2017 13:36:16 -0600
Subject: [PATCH] nvme-cli: fix wrong bitmask in number of rpmb units.
In-Reply-To: <1509213712-32675-1-git-send-email-minwoo.im.dev@gmail.com>
References: <1509213712-32675-1-git-send-email-minwoo.im.dev@gmail.com>
Message-ID: <20171030193615.GH17245@localhost.localdomain>

On Sun, Oct 29, 2017 at 03:01:52AM +0900, Minwoo Im wrote:
> Number of RPMB Units is a 3-bit field[02:00].
> Bitmask should be 0111b(7h) to get this field.
> 
> Signed-off-by: Minwoo Im <minwoo.im.dev at gmail.com>

Applied, thanks.


From james.smart at broadcom.com  Mon Oct 30 12:52:44 2017
From: james.smart at broadcom.com (James Smart)
Date: Mon, 30 Oct 2017 12:52:44 -0700
Subject: [PATCH 2/5] nvme-fc: merge __nvme_fc_schedule_delete_work into
 __nvme_fc_del_ctrl
In-Reply-To: <20171029084431.8702-3-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
 <20171029084431.8702-3-hch@lst.de>
Message-ID: <3df3f145-8586-baef-a30e-ad31bef7a1e2@broadcom.com>

On 10/29/2017 1:44 AM, Christoph Hellwig wrote:
> No need to have two functions doing the same thing.
>
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>   drivers/nvme/host/fc.c | 20 ++++++--------------
>   1 file changed, 6 insertions(+), 14 deletions(-)
>
>

Looks fine.

Reviewed-by: James Smart? <james.smart at broadcom.com>


From james.smart at broadcom.com  Mon Oct 30 12:58:06 2017
From: james.smart at broadcom.com (James Smart)
Date: Mon, 30 Oct 2017 12:58:06 -0700
Subject: [PATCH 3/5] nvme: move controller deletion to common code
In-Reply-To: <20171029084431.8702-4-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
 <20171029084431.8702-4-hch@lst.de>
Message-ID: <89831278-db71-8fb4-b562-3da9ebbb5cd5@broadcom.com>

On 10/29/2017 1:44 AM, Christoph Hellwig wrote:
> Move the ->delete_work and the associated helpers to common code instead
> of duplicating them in every driver.  This also adds the missing reference
> get/put for the loop driver.
>
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>

Looks fine.

Reviewed-by: James Smart? <james.smart at broadcom.com>



From james.smart at broadcom.com  Mon Oct 30 13:00:29 2017
From: james.smart at broadcom.com (James Smart)
Date: Mon, 30 Oct 2017 13:00:29 -0700
Subject: [PATCH 5/5] nvme: consolidate common code from ->reset_work
In-Reply-To: <20171029084431.8702-6-hch@lst.de>
References: <20171029084431.8702-1-hch@lst.de>
 <20171029084431.8702-6-hch@lst.de>
Message-ID: <09da8c25-e855-f260-ac56-cff3de54e9f9@broadcom.com>

On 10/29/2017 1:44 AM, Christoph Hellwig wrote:
> No change in behavior except that the FC code cancels two work items a
> little later now.
>
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>


Looks fine.

Reviewed-by: James Smart? <james.smart at broadcom.com>


From Bart.VanAssche at wdc.com  Mon Oct 30 13:37:09 2017
From: Bart.VanAssche at wdc.com (Bart Van Assche)
Date: Mon, 30 Oct 2017 20:37:09 +0000
Subject: [PATCH v2 6/8] scsi/ipr: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <ce6f2f5b-b4bc-380f-8ac1-1e91e4d6bf24@linux.vnet.ibm.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-7-bart.vanassche@wdc.com>
 <c804098f-5543-395c-1395-5778ef74a61b@suse.de>
 <ce6f2f5b-b4bc-380f-8ac1-1e91e4d6bf24@linux.vnet.ibm.com>
Message-ID: <1509395828.27259.21.camel@wdc.com>

On Wed, 2017-10-18 at 15:57 -0500, Brian King wrote:
> On 10/17/2017 01:19 AM, Hannes Reinecke wrote:
> > On 10/17/2017 12:49 AM, Bart Van Assche wrote:
> > > [ ... ]
> > 
> > Not sure if this is a valid conversion.
> > Originally the driver would allocate a single buffer; with this buffer
> > we have two distinct buffers.
> > Given that this is used to download the microcode I'm not sure if this
> > isn't a hardware-dependent structure which requires a single buffer
> > including the sglist.
> > Brian, can you shed some light here?
> 
> The struct ipr_sglist is not a hardware defined data structure, so on initial
> glance, this should be OK. I'll load it up and give it a try to make sure
> it doesn't break code download.

Hello Brian,

Have you already obtained any test results?

Thanks,

Bart.

From brking at linux.vnet.ibm.com  Mon Oct 30 14:01:48 2017
From: brking at linux.vnet.ibm.com (Brian King)
Date: Mon, 30 Oct 2017 16:01:48 -0500
Subject: [PATCH v2 6/8] scsi/ipr: Use sgl_alloc_order() and
 sgl_free_order()
In-Reply-To: <1509395828.27259.21.camel@wdc.com>
References: <20171016224940.1332-1-bart.vanassche@wdc.com>
 <20171016224940.1332-7-bart.vanassche@wdc.com>
 <c804098f-5543-395c-1395-5778ef74a61b@suse.de>
 <ce6f2f5b-b4bc-380f-8ac1-1e91e4d6bf24@linux.vnet.ibm.com>
 <1509395828.27259.21.camel@wdc.com>
Message-ID: <33515fd2-9639-7981-e344-80cdacd40d32@linux.vnet.ibm.com>

On 10/30/2017 03:37 PM, Bart Van Assche wrote:
> On Wed, 2017-10-18 at 15:57 -0500, Brian King wrote:
>> On 10/17/2017 01:19 AM, Hannes Reinecke wrote:
>>> On 10/17/2017 12:49 AM, Bart Van Assche wrote:
>>>> [ ... ]
>>>
>>> Not sure if this is a valid conversion.
>>> Originally the driver would allocate a single buffer; with this buffer
>>> we have two distinct buffers.
>>> Given that this is used to download the microcode I'm not sure if this
>>> isn't a hardware-dependent structure which requires a single buffer
>>> including the sglist.
>>> Brian, can you shed some light here?
>>
>> The struct ipr_sglist is not a hardware defined data structure, so on initial
>> glance, this should be OK. I'll load it up and give it a try to make sure
>> it doesn't break code download.
> 
> Hello Brian,
> 
> Have you already obtained any test results?

Bart,

Yes. I tried this out on an ipr adapter and it looks fine.

Acked-by: Brian King <brking at linux.vnet.ibm.com>

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center



From bart.vanassche at wdc.com  Mon Oct 30 15:42:05 2017
From: bart.vanassche at wdc.com (Bart Van Assche)
Date: Mon, 30 Oct 2017 15:42:05 -0700
Subject: [PATCH v11 7/7] block, nvme: Introduce blk_mq_req_flags_t
In-Reply-To: <20171030224205.25212-1-bart.vanassche@wdc.com>
References: <20171030224205.25212-1-bart.vanassche@wdc.com>
Message-ID: <20171030224205.25212-8-bart.vanassche@wdc.com>

Several block layer and NVMe core functions accept a combination
of BLK_MQ_REQ_* flags through the 'flags' argument but there is
no verification at compile time whether the right type of block
layer flags is passed. Make it possible for sparse to verify this.
This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bart.vanassche at wdc.com>
Reviewed-by: Hannes Reinecke <hare at suse.com>
Cc: linux-nvme at lists.infradead.org
Cc: Christoph Hellwig <hch at lst.de>
Cc: Johannes Thumshirn <jthumshirn at suse.de>
Cc: Ming Lei <ming.lei at redhat.com>
---
 block/blk-core.c          | 12 ++++++------
 block/blk-mq.c            |  4 ++--
 block/blk-mq.h            |  2 +-
 drivers/nvme/host/core.c  |  5 +++--
 drivers/nvme/host/nvme.h  |  5 +++--
 include/linux/blk-mq.h    | 17 +++++++++++------
 include/linux/blk_types.h |  2 ++
 include/linux/blkdev.h    |  4 ++--
 8 files changed, 30 insertions(+), 21 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index d4dc10bb01e3..0a0fdaa474e3 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -800,7 +800,7 @@ EXPORT_SYMBOL(blk_alloc_queue);
  * @q: request queue pointer
  * @flags: BLK_MQ_REQ_NOWAIT and/or BLK_MQ_REQ_PREEMPT
  */
-int blk_queue_enter(struct request_queue *q, unsigned int flags)
+int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)
 {
 	const bool preempt = flags & BLK_MQ_REQ_PREEMPT;
 
@@ -1227,7 +1227,7 @@ int blk_update_nr_requests(struct request_queue *q, unsigned int nr)
  * Returns request pointer on success, with @q->queue_lock *not held*.
  */
 static struct request *__get_request(struct request_list *rl, unsigned int op,
-				     struct bio *bio, unsigned int flags)
+				     struct bio *bio, blk_mq_req_flags_t flags)
 {
 	struct request_queue *q = rl->q;
 	struct request *rq;
@@ -1410,7 +1410,7 @@ static struct request *__get_request(struct request_list *rl, unsigned int op,
  * Returns request pointer on success, with @q->queue_lock *not held*.
  */
 static struct request *get_request(struct request_queue *q, unsigned int op,
-				   struct bio *bio, unsigned int flags)
+				   struct bio *bio, blk_mq_req_flags_t flags)
 {
 	const bool is_sync = op_is_sync(op);
 	DEFINE_WAIT(wait);
@@ -1460,7 +1460,7 @@ static struct request *get_request(struct request_queue *q, unsigned int op,
 
 /* flags: BLK_MQ_REQ_PREEMPT and/or BLK_MQ_REQ_NOWAIT. */
 static struct request *blk_old_get_request(struct request_queue *q,
-					   unsigned int op, unsigned int flags)
+				unsigned int op, blk_mq_req_flags_t flags)
 {
 	struct request *rq;
 	gfp_t gfp_mask = flags & BLK_MQ_REQ_NOWAIT ? GFP_ATOMIC :
@@ -1497,7 +1497,7 @@ static struct request *blk_old_get_request(struct request_queue *q,
  * @flags: BLK_MQ_REQ_* flags, e.g. BLK_MQ_REQ_NOWAIT.
  */
 struct request *blk_get_request_flags(struct request_queue *q, unsigned int op,
-				      unsigned int flags)
+				      blk_mq_req_flags_t flags)
 {
 	struct request *req;
 
@@ -2293,7 +2293,7 @@ blk_qc_t generic_make_request(struct bio *bio)
 	current->bio_list = bio_list_on_stack;
 	do {
 		struct request_queue *q = bio->bi_disk->queue;
-		unsigned int flags = bio->bi_opf & REQ_NOWAIT ?
+		blk_mq_req_flags_t flags = bio->bi_opf & REQ_NOWAIT ?
 			BLK_MQ_REQ_NOWAIT : 0;
 
 		if (likely(blk_queue_enter(q, flags) == 0)) {
diff --git a/block/blk-mq.c b/block/blk-mq.c
index c6bff60e6b8b..c037b1ad64a7 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -380,7 +380,7 @@ static struct request *blk_mq_get_request(struct request_queue *q,
 }
 
 struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op,
-		unsigned int flags)
+		blk_mq_req_flags_t flags)
 {
 	struct blk_mq_alloc_data alloc_data = { .flags = flags };
 	struct request *rq;
@@ -406,7 +406,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op,
 EXPORT_SYMBOL(blk_mq_alloc_request);
 
 struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
-		unsigned int op, unsigned int flags, unsigned int hctx_idx)
+	unsigned int op, blk_mq_req_flags_t flags, unsigned int hctx_idx)
 {
 	struct blk_mq_alloc_data alloc_data = { .flags = flags };
 	struct request *rq;
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 522b420dedc0..5dcfe4fa5e0d 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -110,7 +110,7 @@ static inline void blk_mq_put_ctx(struct blk_mq_ctx *ctx)
 struct blk_mq_alloc_data {
 	/* input parameter */
 	struct request_queue *q;
-	unsigned int flags;
+	blk_mq_req_flags_t flags;
 	unsigned int shallow_depth;
 
 	/* input & output parameter */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index bb2aad078637..01947cd82b5a 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -292,7 +292,7 @@ static struct nvme_ns *nvme_get_ns_from_disk(struct gendisk *disk)
 }
 
 struct request *nvme_alloc_request(struct request_queue *q,
-		struct nvme_command *cmd, unsigned int flags, int qid)
+		struct nvme_command *cmd, blk_mq_req_flags_t flags, int qid)
 {
 	unsigned op = nvme_is_write(cmd) ? REQ_OP_DRV_OUT : REQ_OP_DRV_IN;
 	struct request *req;
@@ -560,7 +560,8 @@ EXPORT_SYMBOL_GPL(nvme_setup_cmd);
  */
 int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		union nvme_result *result, void *buffer, unsigned bufflen,
-		unsigned timeout, int qid, int at_head, int flags)
+		unsigned timeout, int qid, int at_head,
+		blk_mq_req_flags_t flags)
 {
 	struct request *req;
 	int ret;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index d3f3c4447515..61b25e8c222c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -314,14 +314,15 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl);
 
 #define NVME_QID_ANY -1
 struct request *nvme_alloc_request(struct request_queue *q,
-		struct nvme_command *cmd, unsigned int flags, int qid);
+		struct nvme_command *cmd, blk_mq_req_flags_t flags, int qid);
 blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
 		struct nvme_command *cmd);
 int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		void *buf, unsigned bufflen);
 int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		union nvme_result *result, void *buffer, unsigned bufflen,
-		unsigned timeout, int qid, int at_head, int flags);
+		unsigned timeout, int qid, int at_head,
+		blk_mq_req_flags_t flags);
 int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count);
 void nvme_start_keep_alive(struct nvme_ctrl *ctrl);
 void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 22c7f36745fc..38ecf1340266 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -210,16 +210,21 @@ void blk_mq_free_request(struct request *rq);
 bool blk_mq_can_queue(struct blk_mq_hw_ctx *);
 
 enum {
-	BLK_MQ_REQ_NOWAIT	= (1 << 0), /* return when out of requests */
-	BLK_MQ_REQ_RESERVED	= (1 << 1), /* allocate from reserved pool */
-	BLK_MQ_REQ_INTERNAL	= (1 << 2), /* allocate internal/sched tag */
-	BLK_MQ_REQ_PREEMPT	= (1 << 3), /* set RQF_PREEMPT */
+	/* return when out of requests */
+	BLK_MQ_REQ_NOWAIT	= (__force blk_mq_req_flags_t)(1 << 0),
+	/* allocate from reserved pool */
+	BLK_MQ_REQ_RESERVED	= (__force blk_mq_req_flags_t)(1 << 1),
+	/* allocate internal/sched tag */
+	BLK_MQ_REQ_INTERNAL	= (__force blk_mq_req_flags_t)(1 << 2),
+	/* set RQF_PREEMPT */
+	BLK_MQ_REQ_PREEMPT	= (__force blk_mq_req_flags_t)(1 << 3),
 };
 
 struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op,
-		unsigned int flags);
+		blk_mq_req_flags_t flags);
 struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
-		unsigned int op, unsigned int flags, unsigned int hctx_idx);
+		unsigned int op, blk_mq_req_flags_t flags,
+		unsigned int hctx_idx);
 struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag);
 
 enum {
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 3385c89f402e..cbd908478140 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -162,6 +162,8 @@ struct bio {
  */
 #define BIO_RESET_BITS	BVEC_POOL_OFFSET
 
+typedef __u32 __bitwise blk_mq_req_flags_t;
+
 /*
  * Operations and flags common to the bio and request structures.
  * We use 8 bits for encoding the operation, and the remaining 24 for flags.
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 4f91c6462752..9a7d7e775cd0 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -932,7 +932,7 @@ extern void blk_put_request(struct request *);
 extern void __blk_put_request(struct request_queue *, struct request *);
 extern struct request *blk_get_request_flags(struct request_queue *,
 					     unsigned int op,
-					     unsigned int flags);
+					     blk_mq_req_flags_t flags);
 extern struct request *blk_get_request(struct request_queue *, unsigned int op,
 				       gfp_t gfp_mask);
 extern void blk_requeue_request(struct request_queue *, struct request *);
@@ -956,7 +956,7 @@ extern int scsi_cmd_ioctl(struct request_queue *, struct gendisk *, fmode_t,
 extern int sg_scsi_ioctl(struct request_queue *, struct gendisk *, fmode_t,
 			 struct scsi_ioctl_command __user *);
 
-extern int blk_queue_enter(struct request_queue *q, unsigned int flags);
+extern int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags);
 extern void blk_queue_exit(struct request_queue *q);
 extern void blk_start_queue(struct request_queue *q);
 extern void blk_start_queue_async(struct request_queue *q);
-- 
2.14.2



From sagi at grimberg.me  Tue Oct 31 01:55:19 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 31 Oct 2017 10:55:19 +0200
Subject: [PATCH 0/3] Fix request completion holes
Message-ID: <1509440122-1190-1-git-send-email-sagi@grimberg.me>

We have two holes in nvme-rdma when completing request.

1. We never wait for send work request to complete before completing
a request. It is possible that the HCA retries a send operation (due
to dropped ack) after the nvme cqe has already arrived back to the host.
If we unmap the host buffer upon reception of the cqe, the HCA might
get iommu errors when attempting to access an unmapped host buffer.
We must wait also for the send completion before completing a request,
most of the time it will be before the nvme cqe has arrived back so
we pay only for the extra cq entry processing.

2. We don't wait for the request memory region to be fully invalidated
in case the target didn't invalidate remotely. We must wait for the local
invalidation to complete before completing the request.

Sagi Grimberg (3):
  nvme-rdma: don't suppress send completions
  nvme-rdma: don't complete requests before a send work request has
    completed
  nvme-rdma: wait for local invalidation before completing a request

 drivers/nvme/host/rdma.c | 101 +++++++++++++++++++++++------------------------
 1 file changed, 50 insertions(+), 51 deletions(-)

-- 
2.7.4



From sagi at grimberg.me  Tue Oct 31 01:55:20 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 31 Oct 2017 10:55:20 +0200
Subject: [PATCH 1/3] nvme-rdma: don't suppress send completions
In-Reply-To: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
References: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
Message-ID: <1509440122-1190-2-git-send-email-sagi@grimberg.me>

The entire completions suppress mechanism is currently
broken because the HCA might retry a send operation
(due to dropped ack) after the nvme transaction has completed.

In order to handle this, we signal all send completions (besides
async event which is not racing anything).

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 40 ++++------------------------------------
 1 file changed, 4 insertions(+), 36 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 53ad241493a1..ccbae327fe72 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1001,21 +1001,9 @@ static void nvme_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
 		nvme_rdma_wr_error(cq, wc, "SEND");
 }
 
-/*
- * We want to signal completion at least every queue depth/2.  This returns the
- * largest power of two that is not above half of (queue size + 1) to optimize
- * (avoid divisions).
- */
-static inline bool nvme_rdma_queue_sig_limit(struct nvme_rdma_queue *queue)
-{
-	int limit = 1 << ilog2((queue->queue_size + 1) / 2);
-
-	return (atomic_inc_return(&queue->sig_count) & (limit - 1)) == 0;
-}
-
 static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 		struct nvme_rdma_qe *qe, struct ib_sge *sge, u32 num_sge,
-		struct ib_send_wr *first, bool flush)
+		struct ib_send_wr *first)
 {
 	struct ib_send_wr wr, *bad_wr;
 	int ret;
@@ -1031,24 +1019,7 @@ static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 	wr.sg_list    = sge;
 	wr.num_sge    = num_sge;
 	wr.opcode     = IB_WR_SEND;
-	wr.send_flags = 0;
-
-	/*
-	 * Unsignalled send completions are another giant desaster in the
-	 * IB Verbs spec:  If we don't regularly post signalled sends
-	 * the send queue will fill up and only a QP reset will rescue us.
-	 * Would have been way to obvious to handle this in hardware or
-	 * at least the RDMA stack..
-	 *
-	 * Always signal the flushes. The magic request used for the flush
-	 * sequencer is not allocated in our driver's tagset and it's
-	 * triggered to be freed by blk_cleanup_queue(). So we need to
-	 * always mark it as signaled to ensure that the "wr_cqe", which is
-	 * embedded in request's payload, is not freed when __ib_process_cq()
-	 * calls wr_cqe->done().
-	 */
-	if (nvme_rdma_queue_sig_limit(queue) || flush)
-		wr.send_flags |= IB_SEND_SIGNALED;
+	wr.send_flags = IB_SEND_SIGNALED;
 
 	if (first)
 		first->next = &wr;
@@ -1122,7 +1093,7 @@ static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	ib_dma_sync_single_for_device(dev, sqe->dma, sizeof(*cmd),
 			DMA_TO_DEVICE);
 
-	ret = nvme_rdma_post_send(queue, sqe, &sge, 1, NULL, false);
+	ret = nvme_rdma_post_send(queue, sqe, &sge, 1, NULL);
 	WARN_ON_ONCE(ret);
 }
 
@@ -1407,7 +1378,6 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
 	struct nvme_rdma_qe *sqe = &req->sqe;
 	struct nvme_command *c = sqe->data;
-	bool flush = false;
 	struct ib_device *dev;
 	blk_status_t ret;
 	int err;
@@ -1439,10 +1409,8 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 	ib_dma_sync_single_for_device(dev, sqe->dma,
 			sizeof(struct nvme_command), DMA_TO_DEVICE);
 
-	if (req_op(rq) == REQ_OP_FLUSH)
-		flush = true;
 	err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
-			req->mr->need_inval ? &req->reg_wr.wr : NULL, flush);
+			req->mr->need_inval ? &req->reg_wr.wr : NULL);
 	if (unlikely(err)) {
 		nvme_rdma_unmap_data(queue, rq);
 		goto err;
-- 
2.7.4



From sagi at grimberg.me  Tue Oct 31 01:55:21 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 31 Oct 2017 10:55:21 +0200
Subject: [PATCH 2/3] nvme-rdma: don't complete requests before a send work
 request has completed
In-Reply-To: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
References: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
Message-ID: <1509440122-1190-3-git-send-email-sagi@grimberg.me>

In order to guarantee that the HCA will never get an access violation
(either from invalidated rkey or from iommu) when retrying a send
operation we must complete a request only when both send completion
and the nvme cqe has arrived.

Only then we are safe to invalidate the rkey (if needed), unmap
the host buffers, and complete the IO.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 35 +++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index ccbae327fe72..ae1fb66358f7 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -67,6 +67,9 @@ struct nvme_rdma_request {
 	struct nvme_request	req;
 	struct ib_mr		*mr;
 	struct nvme_rdma_qe	sqe;
+	struct nvme_completion	cqe;
+	bool			send_completed;
+	bool			resp_completed;
 	struct ib_sge		sge[1 + NVME_RDMA_MAX_INLINE_SEGMENTS];
 	u32			num_sge;
 	int			nents;
@@ -961,6 +964,8 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue,
 	req->num_sge = 1;
 	req->inline_data = false;
 	req->mr->need_inval = false;
+	req->send_completed = false;
+	req->resp_completed = false;
 
 	c->common.flags |= NVME_CMD_SGL_METABUF;
 
@@ -997,13 +1002,25 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue,
 
 static void nvme_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
 {
-	if (unlikely(wc->status != IB_WC_SUCCESS))
+	struct nvme_rdma_qe *qe =
+		container_of(wc->wr_cqe, struct nvme_rdma_qe, cqe);
+	struct nvme_rdma_request *req =
+		container_of(qe, struct nvme_rdma_request, sqe);
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (unlikely(wc->status != IB_WC_SUCCESS)) {
 		nvme_rdma_wr_error(cq, wc, "SEND");
+		return;
+	}
+
+	req->send_completed = true;
+	if (req->resp_completed)
+		nvme_end_request(rq, req->cqe.status, req->cqe.result);
 }
 
 static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 		struct nvme_rdma_qe *qe, struct ib_sge *sge, u32 num_sge,
-		struct ib_send_wr *first)
+		struct ib_send_wr *first, bool signal)
 {
 	struct ib_send_wr wr, *bad_wr;
 	int ret;
@@ -1019,7 +1036,7 @@ static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 	wr.sg_list    = sge;
 	wr.num_sge    = num_sge;
 	wr.opcode     = IB_WR_SEND;
-	wr.send_flags = IB_SEND_SIGNALED;
+	wr.send_flags = signal ? IB_SEND_SIGNALED : 0;
 
 	if (first)
 		first->next = &wr;
@@ -1093,7 +1110,7 @@ static void nvme_rdma_submit_async_event(struct nvme_ctrl *arg, int aer_idx)
 	ib_dma_sync_single_for_device(dev, sqe->dma, sizeof(*cmd),
 			DMA_TO_DEVICE);
 
-	ret = nvme_rdma_post_send(queue, sqe, &sge, 1, NULL);
+	ret = nvme_rdma_post_send(queue, sqe, &sge, 1, NULL, false);
 	WARN_ON_ONCE(ret);
 }
 
@@ -1117,11 +1134,17 @@ static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue,
 	if (rq->tag == tag)
 		ret = 1;
 
+	req->cqe.status = cqe->status;
+	req->cqe.result = cqe->result;
+	req->resp_completed = true;
+
 	if ((wc->wc_flags & IB_WC_WITH_INVALIDATE) &&
 	    wc->ex.invalidate_rkey == req->mr->rkey)
 		req->mr->need_inval = false;
 
-	nvme_end_request(rq, cqe->status, cqe->result);
+	if (req->send_completed)
+		nvme_end_request(rq, req->cqe.status, req->cqe.result);
+
 	return ret;
 }
 
@@ -1410,7 +1433,7 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 			sizeof(struct nvme_command), DMA_TO_DEVICE);
 
 	err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
-			req->mr->need_inval ? &req->reg_wr.wr : NULL);
+			req->mr->need_inval ? &req->reg_wr.wr : NULL, true);
 	if (unlikely(err)) {
 		nvme_rdma_unmap_data(queue, rq);
 		goto err;
-- 
2.7.4



From sagi at grimberg.me  Tue Oct 31 01:55:22 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 31 Oct 2017 10:55:22 +0200
Subject: [PATCH 3/3] nvme-rdma: wait for local invalidation before completing
 a request
In-Reply-To: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
References: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
Message-ID: <1509440122-1190-4-git-send-email-sagi@grimberg.me>

We must not complete a request before the host memory region is
invalidated. Luckily we have send with invalidate protocol support
so we usually don't need to execute it, but in case the target
did not invalidate a memory region for us, we must wait for
the invalidation to complete before unmapping host memory and
completing the I/O.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/rdma.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index ae1fb66358f7..b7e0fb0fe913 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -818,8 +818,19 @@ static void nvme_rdma_memreg_done(struct ib_cq *cq, struct ib_wc *wc)
 
 static void nvme_rdma_inv_rkey_done(struct ib_cq *cq, struct ib_wc *wc)
 {
-	if (unlikely(wc->status != IB_WC_SUCCESS))
+	struct nvme_rdma_request *req =
+		container_of(wc->wr_cqe, struct nvme_rdma_request, reg_cqe);
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (unlikely(wc->status != IB_WC_SUCCESS)) {
 		nvme_rdma_wr_error(cq, wc, "LOCAL_INV");
+		return;
+	}
+
+	req->mr->need_inval = false;
+	if (req->resp_completed && req->send_completed)
+		nvme_end_request(rq, req->cqe.status, req->cqe.result);
+
 }
 
 static int nvme_rdma_inv_rkey(struct nvme_rdma_queue *queue,
@@ -830,7 +841,7 @@ static int nvme_rdma_inv_rkey(struct nvme_rdma_queue *queue,
 		.opcode		    = IB_WR_LOCAL_INV,
 		.next		    = NULL,
 		.num_sge	    = 0,
-		.send_flags	    = 0,
+		.send_flags	    = IB_SEND_SIGNALED,
 		.ex.invalidate_rkey = req->mr->rkey,
 	};
 
@@ -844,24 +855,12 @@ static void nvme_rdma_unmap_data(struct nvme_rdma_queue *queue,
 		struct request *rq)
 {
 	struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
-	struct nvme_rdma_ctrl *ctrl = queue->ctrl;
 	struct nvme_rdma_device *dev = queue->device;
 	struct ib_device *ibdev = dev->dev;
-	int res;
 
 	if (!blk_rq_bytes(rq))
 		return;
 
-	if (req->mr->need_inval && test_bit(NVME_RDMA_Q_LIVE, &req->queue->flags)) {
-		res = nvme_rdma_inv_rkey(queue, req);
-		if (unlikely(res < 0)) {
-			dev_err(ctrl->ctrl.device,
-				"Queueing INV WR for rkey %#x failed (%d)\n",
-				req->mr->rkey, res);
-			nvmf_error_recovery(&queue->ctrl->ctrl);
-		}
-	}
-
 	ib_dma_unmap_sg(ibdev, req->sg_table.sgl,
 			req->nents, rq_data_dir(rq) ==
 				    WRITE ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
@@ -1014,7 +1013,7 @@ static void nvme_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
 	}
 
 	req->send_completed = true;
-	if (req->resp_completed)
+	if (req->resp_completed && !req->mr->need_inval)
 		nvme_end_request(rq, req->cqe.status, req->cqe.result);
 }
 
@@ -1139,10 +1138,19 @@ static int nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue,
 	req->resp_completed = true;
 
 	if ((wc->wc_flags & IB_WC_WITH_INVALIDATE) &&
-	    wc->ex.invalidate_rkey == req->mr->rkey)
+	    wc->ex.invalidate_rkey == req->mr->rkey) {
 		req->mr->need_inval = false;
+	} else if (req->mr->need_inval) {
+		ret = nvme_rdma_inv_rkey(queue, req);
+		if (unlikely(ret < 0)) {
+			dev_err(queue->ctrl->ctrl.device,
+				"Queueing INV WR for rkey %#x failed (%d)\n",
+				req->mr->rkey, ret);
+			nvmf_error_recovery(&queue->ctrl->ctrl);
+		}
+	}
 
-	if (req->send_completed)
+	if (req->send_completed && !req->mr->need_inval)
 		nvme_end_request(rq, req->cqe.status, req->cqe.result);
 
 	return ret;
-- 
2.7.4



From maxg at mellanox.com  Tue Oct 31 02:38:25 2017
From: maxg at mellanox.com (Max Gurtovoy)
Date: Tue, 31 Oct 2017 11:38:25 +0200
Subject: [PATCH 0/3] Fix request completion holes
In-Reply-To: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
References: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
Message-ID: <8abcde91-9150-2982-3900-078619bcdac0@mellanox.com>



On 10/31/2017 10:55 AM, Sagi Grimberg wrote:
> We have two holes in nvme-rdma when completing request.
> 

Thanks Sagi.
We'll run these patches in our lab + our patches for the performance 
improvments that we sent few days ago and update the results.

I guess I'll need to run it with my cq moderation patches to get less 
penalty in performance.


From sagi at grimberg.me  Tue Oct 31 04:10:10 2017
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 31 Oct 2017 13:10:10 +0200
Subject: [PATCH 0/3] Fix request completion holes
In-Reply-To: <8abcde91-9150-2982-3900-078619bcdac0@mellanox.com>
References: <1509440122-1190-1-git-send-email-sagi@grimberg.me>
 <8abcde91-9150-2982-3900-078619bcdac0@mellanox.com>
Message-ID: <a950a671-22a1-432c-555e-8309c8a64a88@grimberg.me>

> Thanks Sagi.
> We'll run these patches in our lab + our patches for the performance 
> improvments that we sent few days ago and update the results.

Thanks,

> I guess I'll need to run it with my cq moderation patches to get less 
> penalty in performance.

Fine, but just a heads up, any cq moderation that is not adaptive is not 
something
that we'd like to pursue. I have a generic adaptive moderation library 
in the
works, so any ideas should be integrated into that (or something 
equivalent).


From martin.petersen at oracle.com  Tue Oct 31 05:22:25 2017
From: martin.petersen at oracle.com (Martin K. Petersen)
Date: Tue, 31 Oct 2017 08:22:25 -0400
Subject: [PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller
 rescan
In-Reply-To: <20171029011456.12885-1-jsmart2021@gmail.com> (James Smart's
 message of "Sat, 28 Oct 2017 18:14:51 -0700")
References: <20171029011456.12885-1-jsmart2021@gmail.com>
Message-ID: <yq1wp3bcxry.fsf@oracle.com>


James,

> Note: These patches were posted to the nvme list this morning. I should
> have cc'd the scsi list as well - so I'm not posting to the scsi list.
> I fully expect all patches to be pulled via the nvme-4.15 tree then the
> block tree.

The SCSI pieces look OK to me.

Acked-by: Martin K. Petersen <martin.petersen at oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering


From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:26 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:26 +0530
Subject: [PATCH 00/13] mpt3sas driver NVMe support:
Message-ID: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>

Ventura Series controller are Tri-mode. The controller and
firmware are capable of supporting NVMe devices and
PCIe switches to be connected with the controller. This
patch set adds driver level support for NVMe devices and
PCIe switches.

Suganath Prabu S (13):
  mpt3sas: Add nvme device support in slave alloc,      target  alloc
    and probe
  mpt3sas: SGL to PRP Translation for I/Os to NVMe      devices
  mpt3sas: Added support for nvme encapsulated request      message.
  mpt3sas: API 's to support NVMe drive addition to  SML
  mpt3sas: API's to remove nvme drive from sml
  mpt3sas: Handle NVMe PCIe device related events      generated from
    firmware.
  mpt3sas: Set NVMe device queue depth as 128
  mpt3sas: scan and add nvme device after controller      reset
  mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
  mpt3sas: NVMe drive support for BTDHMAPPING ioctl      command and log
    info
  mpt3sas: Fix nvme drives checking for tlr.
  mpt3sas: Fix sparse warnings
  mpt3sas: Update mpt3sas driver version.

 drivers/scsi/mpt3sas/mpt3sas_base.c      |  643 ++++++++-
 drivers/scsi/mpt3sas/mpt3sas_base.h      |  171 ++-
 drivers/scsi/mpt3sas/mpt3sas_config.c    |  100 ++
 drivers/scsi/mpt3sas/mpt3sas_ctl.c       |  158 ++-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c     | 2080 +++++++++++++++++++++++++++---
 drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |    2 +-
 7 files changed, 3054 insertions(+), 330 deletions(-)

Thanks,
Suganath Prabu S
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:27 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:27 +0530
Subject: [PATCH 01/13] mpt3sas: Add nvme device support in slave alloc,
 target  alloc and probe
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-2-git-send-email-suganath-prabu.subramani@broadcom.com>

1) Added support for probing pcie device and adding NVMe drives to
SML and driver's internal list pcie_device_list.

2) Added support for determing NVMe as boot device.

3) Added nvme device support for call back functions scan_finished
target_alloc,slave_alloc,target destroy and slave destroy.

 a) During scan, pcie devices are probed and added to SML to drivers
internal list.

 b) target_alloc & slave alloc API's allocates resources for
(MPT3SAS_TARGET & MPT3SAS_DEVICE) private datas and holds
information like handle, target_id etc.

 c) slave_destroy & target_destroy are called when driver unregisters
or removes device. Also frees allocated resources and info.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.h      | 110 +++++++-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c     | 431 +++++++++++++++++++++++++++++--
 3 files changed, 647 insertions(+), 124 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 0fe3969..dd04a28 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -161,6 +161,7 @@
 #define MPT_TARGET_FLAGS_VOLUME		0x02
 #define MPT_TARGET_FLAGS_DELETED	0x04
 #define MPT_TARGET_FASTPATH_IO		0x08
+#define MPT_TARGET_FLAGS_PCIE_DEVICE	0x10
 
 #define SAS2_PCI_DEVICE_B0_REVISION	(0x01)
 #define SAS3_PCI_DEVICE_C0_REVISION	(0x02)
@@ -359,7 +360,8 @@ struct Mpi2ManufacturingPage11_t {
  * @flags: MPT_TARGET_FLAGS_XXX flags
  * @deleted: target flaged for deletion
  * @tm_busy: target is busy with TM request.
- * @sdev: The sas_device associated with this target
+ * @sas_dev: The sas_device associated with this target
+ * @pcie_dev: The pcie device associated with this target
  */
 struct MPT3SAS_TARGET {
 	struct scsi_target *starget;
@@ -370,7 +372,8 @@ struct MPT3SAS_TARGET {
 	u32	flags;
 	u8	deleted;
 	u8	tm_busy;
-	struct _sas_device *sdev;
+	struct _sas_device *sas_dev;
+	struct _pcie_device *pcie_dev;
 };
 
 
@@ -514,6 +517,89 @@ static inline void sas_device_put(struct _sas_device *s)
 	kref_put(&s->refcount, sas_device_free);
 }
 
+/*
+ * struct _pcie_device - attached PCIe device information
+ * @list: pcie device list
+ * @starget: starget object
+ * @wwid: device WWID
+ * @handle: device handle
+ * @device_info: bitfield provides detailed info about the device
+ * @id: target id
+ * @channel: target channel
+ * @slot: slot number
+ * @port_num: port number
+ * @responding: used in _scsih_pcie_device_mark_responding
+ * @fast_path: fast path feature enable bit
+ * @nvme_mdts: MaximumDataTransferSize from PCIe Device Page 2 for
+ *		NVMe device only
+ * @enclosure_handle: enclosure handle
+ * @enclosure_logical_id: enclosure logical identifier
+ * @enclosure_level: The level of device's enclosure from the controller
+ * @connector_name: ASCII value of the Connector's name
+ * @serial_number: pointer of serial number string allocated runtime
+ * @refcount: reference count for deletion
+ */
+struct _pcie_device {
+	struct list_head list;
+	struct scsi_target *starget;
+	u64	wwid;
+	u16	handle;
+	u32	device_info;
+	int	id;
+	int	channel;
+	u16	slot;
+	u8	port_num;
+	u8	responding;
+	u8	fast_path;
+	u32	nvme_mdts;
+	u16	enclosure_handle;
+	u64	enclosure_logical_id;
+	u8	enclosure_level;
+	u8	connector_name[4];
+	u8	*serial_number;
+	struct kref refcount;
+};
+/**
+ * pcie_device_get - Increment the pcie device reference count
+ *
+ * @p: pcie_device object
+ *
+ * When ever this function called it will increment the
+ * reference count of the pcie device for which this function called.
+ *
+ */
+static inline void pcie_device_get(struct _pcie_device *p)
+{
+	kref_get(&p->refcount);
+}
+
+/**
+ * pcie_device_free - Release the pcie device object
+ * @r - kref object
+ *
+ * Free's the pcie device object. It will be called when reference count
+ * reaches to zero.
+ */
+static inline void pcie_device_free(struct kref *r)
+{
+	kfree(container_of(r, struct _pcie_device, refcount));
+}
+
+/**
+ * pcie_device_put - Decrement the pcie device reference count
+ *
+ * @p: pcie_device object
+ *
+ * When ever this function called it will decrement the
+ * reference count of the pcie device for which this function called.
+ *
+ * When refernce count reaches to Zero, this will call pcie_device_free to the
+ * pcie_device object.
+ */
+static inline void pcie_device_put(struct _pcie_device *p)
+{
+	kref_put(&p->refcount, pcie_device_free);
+}
 /**
  * struct _raid_device - raid volume link list
  * @list: sas device list
@@ -562,12 +648,13 @@ struct _raid_device {
 
 /**
  * struct _boot_device - boot device info
- * @is_raid: flag to indicate whether this is volume
- * @device: holds pointer for either struct _sas_device or
- *     struct _raid_device
+ *
+ * @channel: sas, raid, or pcie channel
+ * @device: holds pointer for struct _sas_device, struct _raid_device or
+ *     struct _pcie_device
  */
 struct _boot_device {
-	u8 is_raid;
+	int channel;
 	void *device;
 };
 
@@ -831,6 +918,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @bars: bitmask of BAR's that must be configured
  * @mask_interrupts: ignore interrupt
  * @dma_mask: used to set the consistent dma mask
+ * @pci_access_mutex: Mutex to synchronize ioctl,sysfs show path and
+ *			pci resource handling
  * @fault_reset_work_q_name: fw fault work queue
  * @fault_reset_work_q: ""
  * @fault_reset_work: ""
@@ -894,9 +983,13 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @sas_device_list: sas device object list
  * @sas_device_init_list: sas device object list (used only at init time)
  * @sas_device_lock:
+ * @pcie_device_list: pcie device object list
+ * @pcie_device_init_list: pcie device object list (used only at init time)
+ * @pcie_device_lock:
  * @io_missing_delay: time for IO completed by fw when PDR enabled
  * @device_missing_delay: time for device missing by fw when PDR enabled
  * @sas_id : used for setting volume target IDs
+ * @pcie_target_id: used for setting pcie target IDs
  * @blocking_handles: bitmask used to identify which devices need blocking
  * @pd_handles : bitmask for PD handles
  * @pd_handles_sz : size of pd_handle bitmask
@@ -1092,11 +1185,16 @@ struct MPT3SAS_ADAPTER {
 	struct list_head sas_device_list;
 	struct list_head sas_device_init_list;
 	spinlock_t	sas_device_lock;
+	struct list_head pcie_device_list;
+	struct list_head pcie_device_init_list;
+	spinlock_t      pcie_device_lock;
+
 	struct list_head raid_device_list;
 	spinlock_t	raid_device_lock;
 	u8		io_missing_delay;
 	u16		device_missing_delay;
 	int		sas_id;
+	int		pcie_target_id;
 
 	void		*blocking_handles;
 	void		*pd_handles;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index dee8308..4c9b63e 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -60,6 +60,9 @@
 #include "mpt3sas_base.h"
 
 #define RAID_CHANNEL 1
+
+#define PCIE_CHANNEL 2
+
 /* forward proto's */
 static void _scsih_expander_node_remove(struct MPT3SAS_ADAPTER *ioc,
 	struct _sas_node *sas_expander);
@@ -442,21 +445,22 @@ _scsih_get_sas_address(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 /**
  * _scsih_determine_boot_device - determine boot device.
  * @ioc: per adapter object
- * @device: either sas_device or raid_device object
- * @is_raid: [flag] 1 = raid object, 0 = sas object
+ * @device: sas_device or pcie_device object
+ * @channel: SAS or PCIe channel
  *
  * Determines whether this device should be first reported device to
  * to scsi-ml or sas transport, this purpose is for persistent boot device.
  * There are primary, alternate, and current entries in bios page 2. The order
  * priority is primary, alternate, then current.  This routine saves
- * the corresponding device object and is_raid flag in the ioc object.
+ * the corresponding device object.
  * The saved data to be used later in _scsih_probe_boot_devices().
  */
 static void
-_scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc,
-	void *device, u8 is_raid)
+_scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc, void *device,
+	u32 channel)
 {
 	struct _sas_device *sas_device;
+	struct _pcie_device *pcie_device;
 	struct _raid_device *raid_device;
 	u64 sas_address;
 	u64 device_name;
@@ -471,18 +475,24 @@ _scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc,
 	if (!ioc->bios_pg3.BiosVersion)
 		return;
 
-	if (!is_raid) {
-		sas_device = device;
-		sas_address = sas_device->sas_address;
-		device_name = sas_device->device_name;
-		enclosure_logical_id = sas_device->enclosure_logical_id;
-		slot = sas_device->slot;
-	} else {
+	if (channel == RAID_CHANNEL) {
 		raid_device = device;
 		sas_address = raid_device->wwid;
 		device_name = 0;
 		enclosure_logical_id = 0;
 		slot = 0;
+	} else if (channel == PCIE_CHANNEL) {
+		pcie_device = device;
+		sas_address = pcie_device->wwid;
+		device_name = 0;
+		enclosure_logical_id = 0;
+		slot = 0;
+	} else {
+		sas_device = device;
+		sas_address = sas_device->sas_address;
+		device_name = sas_device->device_name;
+		enclosure_logical_id = sas_device->enclosure_logical_id;
+		slot = sas_device->slot;
 	}
 
 	if (!ioc->req_boot_device.device) {
@@ -496,7 +506,7 @@ _scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc,
 			    ioc->name, __func__,
 			    (unsigned long long)sas_address));
 			ioc->req_boot_device.device = device;
-			ioc->req_boot_device.is_raid = is_raid;
+			ioc->req_boot_device.channel = channel;
 		}
 	}
 
@@ -511,7 +521,7 @@ _scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc,
 			    ioc->name, __func__,
 			    (unsigned long long)sas_address));
 			ioc->req_alt_boot_device.device = device;
-			ioc->req_alt_boot_device.is_raid = is_raid;
+			ioc->req_alt_boot_device.channel = channel;
 		}
 	}
 
@@ -526,7 +536,7 @@ _scsih_determine_boot_device(struct MPT3SAS_ADAPTER *ioc,
 			    ioc->name, __func__,
 			    (unsigned long long)sas_address));
 			ioc->current_boot_device.device = device;
-			ioc->current_boot_device.is_raid = is_raid;
+			ioc->current_boot_device.channel = channel;
 		}
 	}
 }
@@ -539,7 +549,7 @@ __mpt3sas_get_sdev_from_target(struct MPT3SAS_ADAPTER *ioc,
 
 	assert_spin_locked(&ioc->sas_device_lock);
 
-	ret = tgt_priv->sdev;
+	ret = tgt_priv->sas_dev;
 	if (ret)
 		sas_device_get(ret);
 
@@ -560,6 +570,44 @@ mpt3sas_get_sdev_from_target(struct MPT3SAS_ADAPTER *ioc,
 	return ret;
 }
 
+static struct _pcie_device *
+__mpt3sas_get_pdev_from_target(struct MPT3SAS_ADAPTER *ioc,
+	struct MPT3SAS_TARGET *tgt_priv)
+{
+	struct _pcie_device *ret;
+
+	assert_spin_locked(&ioc->pcie_device_lock);
+
+	ret = tgt_priv->pcie_dev;
+	if (ret)
+		pcie_device_get(ret);
+
+	return ret;
+}
+
+/**
+ * mpt3sas_get_pdev_from_target - pcie device search
+ * @ioc: per adapter object
+ * @tgt_priv: starget private object
+ *
+ * Context: This function will acquire ioc->pcie_device_lock and will release
+ * before returning the pcie_device object.
+ *
+ * This searches for pcie_device from target, then return pcie_device object.
+ */
+struct _pcie_device *
+mpt3sas_get_pdev_from_target(struct MPT3SAS_ADAPTER *ioc,
+	struct MPT3SAS_TARGET *tgt_priv)
+{
+	struct _pcie_device *ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	ret = __mpt3sas_get_pdev_from_target(ioc, tgt_priv);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	return ret;
+}
 
 struct _sas_device *
 __mpt3sas_get_sdev_by_addr(struct MPT3SAS_ADAPTER *ioc,
@@ -889,6 +937,146 @@ _scsih_sas_device_init_add(struct MPT3SAS_ADAPTER *ioc,
 	spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
 }
 
+
+struct _pcie_device *
+__mpt3sas_get_pdev_by_wwid(struct MPT3SAS_ADAPTER *ioc, u64 wwid)
+{
+	struct _pcie_device *pcie_device;
+
+	assert_spin_locked(&ioc->pcie_device_lock);
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_list, list)
+		if (pcie_device->wwid == wwid)
+			goto found_device;
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_init_list, list)
+		if (pcie_device->wwid == wwid)
+			goto found_device;
+
+	return NULL;
+
+found_device:
+	pcie_device_get(pcie_device);
+	return pcie_device;
+}
+
+
+/**
+ * mpt3sas_get_pdev_by_wwid - pcie device search
+ * @ioc: per adapter object
+ * @wwid: wwid
+ *
+ * Context: This function will acquire ioc->pcie_device_lock and will release
+ * before returning the pcie_device object.
+ *
+ * This searches for pcie_device based on wwid, then return pcie_device object.
+ */
+struct _pcie_device *
+mpt3sas_get_pdev_by_wwid(struct MPT3SAS_ADAPTER *ioc, u64 wwid)
+{
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device = __mpt3sas_get_pdev_by_wwid(ioc, wwid);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	return pcie_device;
+}
+
+
+struct _pcie_device *
+__mpt3sas_get_pdev_by_idchannel(struct MPT3SAS_ADAPTER *ioc, int id,
+	int channel)
+{
+	struct _pcie_device *pcie_device;
+
+	assert_spin_locked(&ioc->pcie_device_lock);
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_list, list)
+		if (pcie_device->id == id && pcie_device->channel == channel)
+			goto found_device;
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_init_list, list)
+		if (pcie_device->id == id && pcie_device->channel == channel)
+			goto found_device;
+
+	return NULL;
+
+found_device:
+	pcie_device_get(pcie_device);
+	return pcie_device;
+}
+
+
+/**
+ * mpt3sas_get_pdev_by_idchannel - pcie device search
+ * @ioc: per adapter object
+ * @id: Target ID
+ * @channel: Channel ID
+ *
+ * Context: This function will acquire ioc->pcie_device_lock and will release
+ * before returning the pcie_device object.
+ *
+ * This searches for pcie_device based on id and channel, then return
+ * pcie_device object.
+ */
+struct _pcie_device *
+mpt3sas_get_pdev_by_idchannel(struct MPT3SAS_ADAPTER *ioc, int id, int channel)
+{
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device = __mpt3sas_get_pdev_by_idchannel(ioc, id, channel);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	return pcie_device;
+}
+/**
+ * _scsih_pcie_device_remove - remove pcie_device from list.
+ * @ioc: per adapter object
+ * @pcie_device: the pcie_device object
+ * Context: This function will acquire ioc->pcie_device_lock.
+ *
+ * If pcie_device is on the list, remove it and decrement its reference count.
+ */
+static void
+_scsih_pcie_device_remove(struct MPT3SAS_ADAPTER *ioc,
+	struct _pcie_device *pcie_device)
+{
+	unsigned long flags;
+	int was_on_pcie_device_list = 0;
+
+	if (!pcie_device)
+		return;
+	pr_info(MPT3SAS_FMT
+		"removing handle(0x%04x), wwid(0x%016llx)\n",
+		ioc->name, pcie_device->handle,
+		(unsigned long long) pcie_device->wwid);
+	if (pcie_device->enclosure_handle != 0)
+		pr_info(MPT3SAS_FMT
+			"removing enclosure logical id(0x%016llx), slot(%d)\n",
+			ioc->name,
+			(unsigned long long)pcie_device->enclosure_logical_id,
+		pcie_device->slot);
+	if (pcie_device->connector_name[0] != '\0')
+		pr_info(MPT3SAS_FMT
+			"removing enclosure level(0x%04x), connector name( %s)\n",
+			ioc->name, pcie_device->enclosure_level,
+			pcie_device->connector_name);
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	if (!list_empty(&pcie_device->list)) {
+		list_del_init(&pcie_device->list);
+		was_on_pcie_device_list = 1;
+	}
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+	if (was_on_pcie_device_list) {
+		kfree(pcie_device->serial_number);
+		pcie_device_put(pcie_device);
+	}
+}
 /**
  * _scsih_raid_device_find_by_id - raid device search
  * @ioc: per adapter object
@@ -1316,6 +1504,7 @@ scsih_target_alloc(struct scsi_target *starget)
 	struct MPT3SAS_TARGET *sas_target_priv_data;
 	struct _sas_device *sas_device;
 	struct _raid_device *raid_device;
+	struct _pcie_device *pcie_device;
 	unsigned long flags;
 	struct sas_rphy *rphy;
 
@@ -1345,6 +1534,28 @@ scsih_target_alloc(struct scsi_target *starget)
 		return 0;
 	}
 
+	/* PCIe devices */
+	if (starget->channel == PCIE_CHANNEL) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_by_idchannel(ioc, starget->id,
+			starget->channel);
+		if (pcie_device) {
+			sas_target_priv_data->handle = pcie_device->handle;
+			sas_target_priv_data->sas_address = pcie_device->wwid;
+			sas_target_priv_data->pcie_dev = pcie_device;
+			pcie_device->starget = starget;
+			pcie_device->id = starget->id;
+			pcie_device->channel = starget->channel;
+			sas_target_priv_data->flags |=
+				MPT_TARGET_FLAGS_PCIE_DEVICE;
+			if (pcie_device->fast_path)
+				sas_target_priv_data->flags |=
+					MPT_TARGET_FASTPATH_IO;
+		}
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		return 0;
+	}
+
 	/* sas/sata devices */
 	spin_lock_irqsave(&ioc->sas_device_lock, flags);
 	rphy = dev_to_rphy(starget->dev.parent);
@@ -1354,7 +1565,7 @@ scsih_target_alloc(struct scsi_target *starget)
 	if (sas_device) {
 		sas_target_priv_data->handle = sas_device->handle;
 		sas_target_priv_data->sas_address = sas_device->sas_address;
-		sas_target_priv_data->sdev = sas_device;
+		sas_target_priv_data->sas_dev = sas_device;
 		sas_device->starget = starget;
 		sas_device->id = starget->id;
 		sas_device->channel = starget->channel;
@@ -1362,7 +1573,8 @@ scsih_target_alloc(struct scsi_target *starget)
 			sas_target_priv_data->flags |=
 			    MPT_TARGET_FLAGS_RAID_COMPONENT;
 		if (sas_device->fast_path)
-			sas_target_priv_data->flags |= MPT_TARGET_FASTPATH_IO;
+			sas_target_priv_data->flags |=
+					MPT_TARGET_FASTPATH_IO;
 	}
 	spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
 
@@ -1383,7 +1595,9 @@ scsih_target_destroy(struct scsi_target *starget)
 	struct MPT3SAS_TARGET *sas_target_priv_data;
 	struct _sas_device *sas_device;
 	struct _raid_device *raid_device;
+	struct _pcie_device *pcie_device;
 	unsigned long flags;
+	struct sas_rphy *rphy;
 
 	sas_target_priv_data = starget->hostdata;
 	if (!sas_target_priv_data)
@@ -1401,7 +1615,29 @@ scsih_target_destroy(struct scsi_target *starget)
 		goto out;
 	}
 
+	if (starget->channel == PCIE_CHANNEL) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_from_target(ioc,
+							sas_target_priv_data);
+		if (pcie_device && (pcie_device->starget == starget) &&
+			(pcie_device->id == starget->id) &&
+			(pcie_device->channel == starget->channel))
+			pcie_device->starget = NULL;
+
+		if (pcie_device) {
+			/*
+			 * Corresponding get() is in _scsih_target_alloc()
+			 */
+			sas_target_priv_data->pcie_dev = NULL;
+			pcie_device_put(pcie_device);
+			pcie_device_put(pcie_device);
+		}
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		goto out;
+	}
+
 	spin_lock_irqsave(&ioc->sas_device_lock, flags);
+	rphy = dev_to_rphy(starget->dev.parent);
 	sas_device = __mpt3sas_get_sdev_from_target(ioc, sas_target_priv_data);
 	if (sas_device && (sas_device->starget == starget) &&
 	    (sas_device->id == starget->id) &&
@@ -1412,7 +1648,7 @@ scsih_target_destroy(struct scsi_target *starget)
 		/*
 		 * Corresponding get() is in _scsih_target_alloc()
 		 */
-		sas_target_priv_data->sdev = NULL;
+		sas_target_priv_data->sas_dev = NULL;
 		sas_device_put(sas_device);
 
 		sas_device_put(sas_device);
@@ -1441,6 +1677,7 @@ scsih_slave_alloc(struct scsi_device *sdev)
 	struct scsi_target *starget;
 	struct _raid_device *raid_device;
 	struct _sas_device *sas_device;
+	struct _pcie_device *pcie_device;
 	unsigned long flags;
 
 	sas_device_priv_data = kzalloc(sizeof(*sas_device_priv_data),
@@ -1469,8 +1706,22 @@ scsih_slave_alloc(struct scsi_device *sdev)
 			raid_device->sdev = sdev; /* raid is single lun */
 		spin_unlock_irqrestore(&ioc->raid_device_lock, flags);
 	}
+	if (starget->channel == PCIE_CHANNEL) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_by_wwid(ioc,
+				sas_target_priv_data->sas_address);
+		if (pcie_device && (pcie_device->starget == NULL)) {
+			sdev_printk(KERN_INFO, sdev,
+			    "%s : pcie_device->starget set to starget @ %d\n",
+			    __func__, __LINE__);
+			pcie_device->starget = starget;
+		}
+
+		if (pcie_device)
+			pcie_device_put(pcie_device);
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
 
-	if (!(sas_target_priv_data->flags & MPT_TARGET_FLAGS_VOLUME)) {
+	} else  if (!(sas_target_priv_data->flags & MPT_TARGET_FLAGS_VOLUME)) {
 		spin_lock_irqsave(&ioc->sas_device_lock, flags);
 		sas_device = __mpt3sas_get_sdev_by_addr(ioc,
 					sas_target_priv_data->sas_address);
@@ -1504,6 +1755,7 @@ scsih_slave_destroy(struct scsi_device *sdev)
 	struct Scsi_Host *shost;
 	struct MPT3SAS_ADAPTER *ioc;
 	struct _sas_device *sas_device;
+	struct _pcie_device *pcie_device;
 	unsigned long flags;
 
 	if (!sdev->hostdata)
@@ -1516,7 +1768,19 @@ scsih_slave_destroy(struct scsi_device *sdev)
 	shost = dev_to_shost(&starget->dev);
 	ioc = shost_priv(shost);
 
-	if (!(sas_target_priv_data->flags & MPT_TARGET_FLAGS_VOLUME)) {
+	if (sas_target_priv_data->flags & MPT_TARGET_FLAGS_PCIE_DEVICE) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_from_target(ioc,
+				sas_target_priv_data);
+		if (pcie_device && !sas_target_priv_data->num_luns)
+			pcie_device->starget = NULL;
+
+		if (pcie_device)
+			pcie_device_put(pcie_device);
+
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	} else if (!(sas_target_priv_data->flags & MPT_TARGET_FLAGS_VOLUME)) {
 		spin_lock_irqsave(&ioc->sas_device_lock, flags);
 		sas_device = __mpt3sas_get_sdev_from_target(ioc,
 				sas_target_priv_data);
@@ -1619,6 +1883,7 @@ scsih_get_resync(struct device *dev)
 
 	percent_complete = 0;
 	handle = 0;
+
 	if (ioc->is_warpdrive)
 		goto out;
 
@@ -8398,42 +8663,52 @@ scsih_shutdown(struct pci_dev *pdev)
 static void
 _scsih_probe_boot_devices(struct MPT3SAS_ADAPTER *ioc)
 {
-	u8 is_raid;
+	u32 channel;
 	void *device;
 	struct _sas_device *sas_device;
 	struct _raid_device *raid_device;
+	struct _pcie_device *pcie_device;
 	u16 handle;
 	u64 sas_address_parent;
 	u64 sas_address;
 	unsigned long flags;
 	int rc;
+	int tid;
 
 	 /* no Bios, return immediately */
 	if (!ioc->bios_pg3.BiosVersion)
 		return;
 
 	device = NULL;
-	is_raid = 0;
 	if (ioc->req_boot_device.device) {
 		device =  ioc->req_boot_device.device;
-		is_raid = ioc->req_boot_device.is_raid;
+		channel = ioc->req_boot_device.channel;
 	} else if (ioc->req_alt_boot_device.device) {
 		device =  ioc->req_alt_boot_device.device;
-		is_raid = ioc->req_alt_boot_device.is_raid;
+		channel = ioc->req_alt_boot_device.channel;
 	} else if (ioc->current_boot_device.device) {
 		device =  ioc->current_boot_device.device;
-		is_raid = ioc->current_boot_device.is_raid;
+		channel = ioc->current_boot_device.channel;
 	}
 
 	if (!device)
 		return;
 
-	if (is_raid) {
+	if (channel == RAID_CHANNEL) {
 		raid_device = device;
 		rc = scsi_add_device(ioc->shost, RAID_CHANNEL,
 		    raid_device->id, 0);
 		if (rc)
 			_scsih_raid_device_remove(ioc, raid_device);
+	} else if (channel == PCIE_CHANNEL) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = device;
+		tid = pcie_device->id;
+		list_move_tail(&pcie_device->list, &ioc->pcie_device_list);
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		rc = scsi_add_device(ioc->shost, PCIE_CHANNEL, tid, 0);
+		if (rc)
+			_scsih_pcie_device_remove(ioc, pcie_device);
 	} else {
 		spin_lock_irqsave(&ioc->sas_device_lock, flags);
 		sas_device = device;
@@ -8566,6 +8841,101 @@ _scsih_probe_sas(struct MPT3SAS_ADAPTER *ioc)
 }
 
 /**
+ * get_next_pcie_device - Get the next pcie device
+ * @ioc: per adapter object
+ *
+ * Get the next pcie device from pcie_device_init_list list.
+ *
+ * Returns pcie device structure if pcie_device_init_list list is not empty
+ * otherwise returns NULL
+ */
+static struct _pcie_device *get_next_pcie_device(struct MPT3SAS_ADAPTER *ioc)
+{
+	struct _pcie_device *pcie_device = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	if (!list_empty(&ioc->pcie_device_init_list)) {
+		pcie_device = list_first_entry(&ioc->pcie_device_init_list,
+				struct _pcie_device, list);
+		pcie_device_get(pcie_device);
+	}
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	return pcie_device;
+}
+
+/**
+ * pcie_device_make_active - Add pcie device to pcie_device_list list
+ * @ioc: per adapter object
+ * @pcie_device: pcie device object
+ *
+ * Add the pcie device which has registered with SCSI Transport Later to
+ * pcie_device_list list
+ */
+static void pcie_device_make_active(struct MPT3SAS_ADAPTER *ioc,
+		struct _pcie_device *pcie_device)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+
+	if (!list_empty(&pcie_device->list)) {
+		list_del_init(&pcie_device->list);
+		pcie_device_put(pcie_device);
+	}
+	pcie_device_get(pcie_device);
+	list_add_tail(&pcie_device->list, &ioc->pcie_device_list);
+
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+}
+
+/**
+ * _scsih_probe_pcie - reporting PCIe devices to scsi-ml
+ * @ioc: per adapter object
+ *
+ * Called during initial loading of the driver.
+ */
+static void
+_scsih_probe_pcie(struct MPT3SAS_ADAPTER *ioc)
+{
+	struct _pcie_device *pcie_device;
+	int rc;
+
+	/* PCIe Device List */
+	while ((pcie_device = get_next_pcie_device(ioc))) {
+		if (pcie_device->starget) {
+			pcie_device_put(pcie_device);
+			continue;
+		}
+		rc = scsi_add_device(ioc->shost, PCIE_CHANNEL,
+			pcie_device->id, 0);
+		if (rc) {
+			_scsih_pcie_device_remove(ioc, pcie_device);
+			pcie_device_put(pcie_device);
+			continue;
+		} else if (!pcie_device->starget) {
+			/* CQ 206770:
+			 * When asyn scanning is enabled, its not possible to
+			 * remove devices while scanning is turned on due to an
+			 * oops in scsi_sysfs_add_sdev()->add_device()->
+			 * sysfs_addrm_start()
+			 */
+			if (!ioc->is_driver_loading) {
+			/* TODO-- Need to find out whether this condition will
+			 * occur or not
+			 */
+				_scsih_pcie_device_remove(ioc, pcie_device);
+				pcie_device_put(pcie_device);
+				continue;
+			}
+		}
+		pcie_device_make_active(ioc, pcie_device);
+		pcie_device_put(pcie_device);
+	}
+}
+
+/**
  * _scsih_probe_devices - probing for devices
  * @ioc: per adapter object
  *
@@ -8593,8 +8963,10 @@ _scsih_probe_devices(struct MPT3SAS_ADAPTER *ioc)
 			_scsih_probe_sas(ioc);
 			_scsih_probe_raid(ioc);
 		}
-	} else
+	} else {
 		_scsih_probe_sas(ioc);
+		_scsih_probe_pcie(ioc);
+	}
 }
 
 /**
@@ -8937,11 +9309,14 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	spin_lock_init(&ioc->sas_node_lock);
 	spin_lock_init(&ioc->fw_event_lock);
 	spin_lock_init(&ioc->raid_device_lock);
+	spin_lock_init(&ioc->pcie_device_lock);
 	spin_lock_init(&ioc->diag_trigger_lock);
 
 	INIT_LIST_HEAD(&ioc->sas_device_list);
 	INIT_LIST_HEAD(&ioc->sas_device_init_list);
 	INIT_LIST_HEAD(&ioc->sas_expander_list);
+	INIT_LIST_HEAD(&ioc->pcie_device_list);
+	INIT_LIST_HEAD(&ioc->pcie_device_init_list);
 	INIT_LIST_HEAD(&ioc->fw_event_list);
 	INIT_LIST_HEAD(&ioc->raid_device_list);
 	INIT_LIST_HEAD(&ioc->sas_hba.sas_port_list);
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:29 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:29 +0530
Subject: [PATCH 03/13] mpt3sas: Added support for nvme encapsulated request
 message.
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-4-git-send-email-suganath-prabu.subramani@broadcom.com>

* Mpt3sas driver uses the NVMe Encapsulated Request message to
send an NVMe command to an NVMe device attached to the IOC.

* Normal I/O commands like reads and writes are passed to the
controller as SCSI commands and the controller has the ability
to translate the commands to NVMe equivalent.

* This encapsulated NVMe command is used by applications to send
direct NVMe commands to NVMe drives.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 276 +++++++++++++++++++++++++++++++++++-
 drivers/scsi/mpt3sas/mpt3sas_base.h |   4 +
 drivers/scsi/mpt3sas/mpt3sas_ctl.c  |  69 ++++++++-
 3 files changed, 342 insertions(+), 7 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 1ad3cbb..b0a75c6 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -557,6 +557,11 @@ _base_sas_ioc_info(struct MPT3SAS_ADAPTER *ioc, MPI2DefaultReply_t *mpi_reply,
 		frame_sz = sizeof(Mpi2SmpPassthroughRequest_t) + ioc->sge_size;
 		func_str = "smp_passthru";
 		break;
+	case MPI2_FUNCTION_NVME_ENCAPSULATED:
+		frame_sz = sizeof(Mpi26NVMeEncapsulatedRequest_t) +
+		    ioc->sge_size;
+		func_str = "nvme_encapsulated";
+		break;
 	default:
 		frame_sz = 32;
 		func_str = "unknown";
@@ -985,7 +990,9 @@ _base_interrupt(int irq, void *bus_id)
 		if (request_desript_type ==
 		    MPI25_RPY_DESCRIPT_FLAGS_FAST_PATH_SCSI_IO_SUCCESS ||
 		    request_desript_type ==
-		    MPI2_RPY_DESCRIPT_FLAGS_SCSI_IO_SUCCESS) {
+		    MPI2_RPY_DESCRIPT_FLAGS_SCSI_IO_SUCCESS ||
+		    request_desript_type ==
+		    MPI26_RPY_DESCRIPT_FLAGS_PCIE_ENCAPSULATED_SUCCESS) {
 			cb_idx = _base_get_cb_idx(ioc, smid);
 			if ((likely(cb_idx < MPT_MAX_CALLBACKS)) &&
 			    (likely(mpt_callbacks[cb_idx] != NULL))) {
@@ -1345,6 +1352,225 @@ _base_build_sg(struct MPT3SAS_ADAPTER *ioc, void *psge,
 	}
 }
 
+/* IEEE format sgls */
+
+/**
+ * _base_build_nvme_prp - This function is called for NVMe end devices to build
+ * a native SGL (NVMe PRP). The native SGL is built starting in the first PRP
+ * entry of the NVMe message (PRP1).  If the data buffer is small enough to be
+ * described entirely using PRP1, then PRP2 is not used.  If needed, PRP2 is
+ * used to describe a larger data buffer.  If the data buffer is too large to
+ * describe using the two PRP entriess inside the NVMe message, then PRP1
+ * describes the first data memory segment, and PRP2 contains a pointer to a PRP
+ * list located elsewhere in memory to describe the remaining data memory
+ * segments.  The PRP list will be contiguous.
+
+ * The native SGL for NVMe devices is a Physical Region Page (PRP).  A PRP
+ * consists of a list of PRP entries to describe a number of noncontigous
+ * physical memory segments as a single memory buffer, just as a SGL does.  Note
+ * however, that this function is only used by the IOCTL call, so the memory
+ * given will be guaranteed to be contiguous.  There is no need to translate
+ * non-contiguous SGL into a PRP in this case.  All PRPs will describe
+ * contiguous space that is one page size each.
+ *
+ * Each NVMe message contains two PRP entries.  The first (PRP1) either contains
+ * a PRP list pointer or a PRP element, depending upon the command.  PRP2
+ * contains the second PRP element if the memory being described fits within 2
+ * PRP entries, or a PRP list pointer if the PRP spans more than two entries.
+ *
+ * A PRP list pointer contains the address of a PRP list, structured as a linear
+ * array of PRP entries.  Each PRP entry in this list describes a segment of
+ * physical memory.
+ *
+ * Each 64-bit PRP entry comprises an address and an offset field.  The address
+ * always points at the beginning of a 4KB physical memory page, and the offset
+ * describes where within that 4KB page the memory segment begins.  Only the
+ * first element in a PRP list may contain a non-zero offest, implying that all
+ * memory segments following the first begin at the start of a 4KB page.
+ *
+ * Each PRP element normally describes 4KB of physical memory, with exceptions
+ * for the first and last elements in the list.  If the memory being described
+ * by the list begins at a non-zero offset within the first 4KB page, then the
+ * first PRP element will contain a non-zero offset indicating where the region
+ * begins within the 4KB page.  The last memory segment may end before the end
+ * of the 4KB segment, depending upon the overall size of the memory being
+ * described by the PRP list.
+ *
+ * Since PRP entries lack any indication of size, the overall data buffer length
+ * is used to determine where the end of the data memory buffer is located, and
+ * how many PRP entries are required to describe it.
+ *
+ * @ioc: per adapter object
+ * @smid: system request message index for getting asscociated SGL
+ * @nvme_encap_request: the NVMe request msg frame pointer
+ * @data_out_dma: physical address for WRITES
+ * @data_out_sz: data xfer size for WRITES
+ * @data_in_dma: physical address for READS
+ * @data_in_sz: data xfer size for READS
+ *
+ * Returns nothing.
+ */
+static void
+_base_build_nvme_prp(struct MPT3SAS_ADAPTER *ioc, u16 smid,
+	Mpi26NVMeEncapsulatedRequest_t *nvme_encap_request,
+	dma_addr_t data_out_dma, size_t data_out_sz, dma_addr_t data_in_dma,
+	size_t data_in_sz)
+{
+	int		prp_size = NVME_PRP_SIZE;
+	u64		*prp_entry, *prp1_entry, *prp2_entry, *prp_entry_phys;
+	u64		*prp_page, *prp_page_phys;
+	u32		offset, entry_len;
+	u32		page_mask_result, page_mask;
+	dma_addr_t	paddr;
+	size_t		length;
+
+	/*
+	 * Not all commands require a data transfer. If no data, just return
+	 * without constructing any PRP.
+	 */
+	if (!data_in_sz && !data_out_sz)
+		return;
+	/*
+	 * Set pointers to PRP1 and PRP2, which are in the NVMe command.
+	 * PRP1 is located at a 24 byte offset from the start of the NVMe
+	 * command.  Then set the current PRP entry pointer to PRP1.
+	 */
+	prp1_entry = (u64 *)(nvme_encap_request->NVMe_Command +
+	    NVME_CMD_PRP1_OFFSET);
+	prp2_entry = (u64 *)(nvme_encap_request->NVMe_Command +
+	    NVME_CMD_PRP2_OFFSET);
+	prp_entry = prp1_entry;
+	/*
+	 * For the PRP entries, use the specially allocated buffer of
+	 * contiguous memory.
+	 */
+	prp_page = (u64 *)mpt3sas_base_get_pcie_sgl(ioc, smid);
+	prp_page_phys = (u64 *)mpt3sas_base_get_pcie_sgl_dma(ioc, smid);
+
+	/*
+	 * Check if we are within 1 entry of a page boundary we don't
+	 * want our first entry to be a PRP List entry.
+	 */
+	page_mask = ioc->page_size - 1;
+	page_mask_result = (uintptr_t)((u8 *)prp_page + prp_size) & page_mask;
+	if (!page_mask_result) {
+		/* Bump up to next page boundary. */
+		prp_page = (u64 *)((u8 *)prp_page + prp_size);
+		prp_page_phys = (u64 *)((u8 *)prp_page_phys + prp_size);
+	}
+
+	/*
+	 * Set PRP physical pointer, which initially points to the current PRP
+	 * DMA memory page.
+	 */
+	prp_entry_phys = prp_page_phys;
+
+	/* Get physical address and length of the data buffer. */
+	if (data_in_sz) {
+		paddr = data_in_dma;
+		length = data_in_sz;
+	} else {
+		paddr = data_out_dma;
+		length = data_out_sz;
+	}
+
+	/* Loop while the length is not zero. */
+	while (length) {
+		/*
+		 * Check if we need to put a list pointer here if we are at
+		 * page boundary - prp_size (8 bytes).
+		 */
+		page_mask_result =
+		    (uintptr_t)((u8 *)prp_entry_phys + prp_size) & page_mask;
+		if (!page_mask_result) {
+			/*
+			 * This is the last entry in a PRP List, so we need to
+			 * put a PRP list pointer here.  What this does is:
+			 *   - bump the current memory pointer to the next
+			 *     address, which will be the next full page.
+			 *   - set the PRP Entry to point to that page.  This
+			 *     is now the PRP List pointer.
+			 *   - bump the PRP Entry pointer the start of the
+			 *     next page.  Since all of this PRP memory is
+			 *     contiguous, no need to get a new page - it's
+			 *     just the next address.
+			 */
+			prp_entry_phys++;
+			*prp_entry = cpu_to_le64((uintptr_t)prp_entry_phys);
+			prp_entry++;
+		}
+
+		/* Need to handle if entry will be part of a page. */
+		offset = (u32)paddr & page_mask;
+		entry_len = ioc->page_size - offset;
+
+		if (prp_entry == prp1_entry) {
+			/*
+			 * Must fill in the first PRP pointer (PRP1) before
+			 * moving on.
+			 */
+			*prp1_entry = cpu_to_le64((u64)paddr);
+
+			/*
+			 * Now point to the second PRP entry within the
+			 * command (PRP2).
+			 */
+			prp_entry = prp2_entry;
+		} else if (prp_entry == prp2_entry) {
+			/*
+			 * Should the PRP2 entry be a PRP List pointer or just
+			 * a regular PRP pointer?  If there is more than one
+			 * more page of data, must use a PRP List pointer.
+			 */
+			if (length > ioc->page_size) {
+				/*
+				 * PRP2 will contain a PRP List pointer because
+				 * more PRP's are needed with this command. The
+				 * list will start at the beginning of the
+				 * contiguous buffer.
+				 */
+				*prp2_entry =
+				    cpu_to_le64((uintptr_t)prp_entry_phys);
+
+				/*
+				 * The next PRP Entry will be the start of the
+				 * first PRP List.
+				 */
+				prp_entry = prp_page;
+			} else {
+				/*
+				 * After this, the PRP Entries are complete.
+				 * This command uses 2 PRP's and no PRP list.
+				 */
+				*prp2_entry = cpu_to_le64((u64)paddr);
+			}
+		} else {
+			/*
+			 * Put entry in list and bump the addresses.
+			 *
+			 * After PRP1 and PRP2 are filled in, this will fill in
+			 * all remaining PRP entries in a PRP List, one per
+			 * each time through the loop.
+			 */
+			*prp_entry = cpu_to_le64((u64)paddr);
+			prp_entry++;
+			prp_entry_phys++;
+		}
+
+		/*
+		 * Bump the phys address of the command's data buffer by the
+		 * entry_len.
+		 */
+		paddr += entry_len;
+
+		/* Decrement length accounting for last partial page. */
+		if (entry_len > length)
+			length = 0;
+		else
+			length -= entry_len;
+	}
+}
+
 /**
  * base_make_prp_nvme -
  * Prepare PRPs(Physical Region Page)- SGLs specific to NVMe drives only
@@ -2794,6 +3020,30 @@ _base_put_smid_hi_priority(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 }
 
 /**
+ * _base_put_smid_nvme_encap - send NVMe encapsulated request to
+ *  firmware
+ * @ioc: per adapter object
+ * @smid: system request message index
+ *
+ * Return nothing.
+ */
+static void
+_base_put_smid_nvme_encap(struct MPT3SAS_ADAPTER *ioc, u16 smid)
+{
+	Mpi2RequestDescriptorUnion_t descriptor;
+	u64 *request = (u64 *)&descriptor;
+
+	descriptor.Default.RequestFlags =
+		MPI26_REQ_DESCRIPT_FLAGS_PCIE_ENCAPSULATED;
+	descriptor.Default.MSIxIndex =  _base_get_msix_index(ioc);
+	descriptor.Default.SMID = cpu_to_le16(smid);
+	descriptor.Default.LMID = 0;
+	descriptor.Default.DescriptorTypeDependent = 0;
+	_base_writeq(*request, &ioc->chip->RequestDescriptorPostLow,
+	    &ioc->scsi_lookup_lock);
+}
+
+/**
  * _base_put_smid_default - Default, primarily used for config pages
  * @ioc: per adapter object
  * @smid: system request message index
@@ -2884,6 +3134,27 @@ _base_put_smid_hi_priority_atomic(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 }
 
 /**
+ * _base_put_smid_nvme_encap_atomic - send NVMe encapsulated request to
+ *   firmware using Atomic Request Descriptor
+ * @ioc: per adapter object
+ * @smid: system request message index
+ *
+ * Return nothing.
+ */
+static void
+_base_put_smid_nvme_encap_atomic(struct MPT3SAS_ADAPTER *ioc, u16 smid)
+{
+	Mpi26AtomicRequestDescriptor_t descriptor;
+	u32 *request = (u32 *)&descriptor;
+
+	descriptor.RequestFlags = MPI26_REQ_DESCRIPT_FLAGS_PCIE_ENCAPSULATED;
+	descriptor.MSIxIndex = _base_get_msix_index(ioc);
+	descriptor.SMID = cpu_to_le16(smid);
+
+	writel(cpu_to_le32(*request), &ioc->chip->AtomicRequestDescriptorPost);
+}
+
+/**
  * _base_put_smid_default - Default, primarily used for config pages
  * use Atomic Request Descriptor
  * @ioc: per adapter object
@@ -5707,6 +5978,7 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc)
 		 */
 		ioc->build_sg_scmd = &_base_build_sg_scmd_ieee;
 		ioc->build_sg = &_base_build_sg_ieee;
+		ioc->build_nvme_prp = &_base_build_nvme_prp;
 		ioc->build_zero_len_sge = &_base_build_zero_len_sge_ieee;
 		ioc->sge_size_ieee = sizeof(Mpi2IeeeSgeSimple64_t);
 
@@ -5718,11 +5990,13 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc)
 		ioc->put_smid_scsi_io = &_base_put_smid_scsi_io_atomic;
 		ioc->put_smid_fast_path = &_base_put_smid_fast_path_atomic;
 		ioc->put_smid_hi_priority = &_base_put_smid_hi_priority_atomic;
+		ioc->put_smid_nvme_encap = &_base_put_smid_nvme_encap_atomic;
 	} else {
 		ioc->put_smid_default = &_base_put_smid_default;
 		ioc->put_smid_scsi_io = &_base_put_smid_scsi_io;
 		ioc->put_smid_fast_path = &_base_put_smid_fast_path;
 		ioc->put_smid_hi_priority = &_base_put_smid_hi_priority;
+		ioc->put_smid_nvme_encap = &_base_put_smid_nvme_encap;
 	}
 
 
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 4758729..034b34d 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1184,6 +1184,9 @@ struct MPT3SAS_ADAPTER {
 	MPT_BUILD_SG    build_sg_mpi;
 	MPT_BUILD_ZERO_LEN_SGE build_zero_len_sge_mpi;
 
+	/* function ptr for NVMe PRP elements only */
+	NVME_BUILD_PRP  build_nvme_prp;
+
 	/* event log */
 	u32		event_type[MPI2_EVENT_NOTIFY_EVENTMASK_WORDS];
 	u32		event_context;
@@ -1354,6 +1357,7 @@ struct MPT3SAS_ADAPTER {
 	PUT_SMID_IO_FP_HIP put_smid_fast_path;
 	PUT_SMID_IO_FP_HIP put_smid_hi_priority;
 	PUT_SMID_DEFAULT put_smid_default;
+	PUT_SMID_DEFAULT put_smid_nvme_encap;
 
 };
 
diff --git a/drivers/scsi/mpt3sas/mpt3sas_ctl.c b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
index 67c7280..205c443 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_ctl.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
@@ -272,6 +272,7 @@ mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index,
 {
 	MPI2DefaultReply_t *mpi_reply;
 	Mpi2SCSIIOReply_t *scsiio_reply;
+	Mpi26NVMeEncapsulatedErrorReply_t *nvme_error_reply;
 	const void *sense_data;
 	u32 sz;
 
@@ -298,6 +299,18 @@ mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index,
 				memcpy(ioc->ctl_cmds.sense, sense_data, sz);
 			}
 		}
+		/*
+		 * Get Error Response data for NVMe device. The ctl_cmds.sense
+		 * buffer is used to store the Error Response data.
+		 */
+		if (mpi_reply->Function == MPI2_FUNCTION_NVME_ENCAPSULATED) {
+			nvme_error_reply =
+			    (Mpi26NVMeEncapsulatedErrorReply_t *)mpi_reply;
+			sz = min_t(u32, NVME_ERROR_RESPONSE_SIZE,
+			    le32_to_cpu(nvme_error_reply->ErrorResponseCount));
+			sense_data = mpt3sas_base_get_sense_buffer(ioc, smid);
+			memcpy(ioc->ctl_cmds.sense, sense_data, sz);
+		}
 	}
 
 	_ctl_display_some_debug(ioc, smid, "ctl_done", mpi_reply);
@@ -641,11 +654,12 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg,
 {
 	MPI2RequestHeader_t *mpi_request = NULL, *request;
 	MPI2DefaultReply_t *mpi_reply;
+	Mpi26NVMeEncapsulatedRequest_t *nvme_encap_request = NULL;
 	u32 ioc_state;
 	u16 smid;
 	unsigned long timeout;
 	u8 issue_reset;
-	u32 sz;
+	u32 sz, sz_arg;
 	void *psge;
 	void *data_out = NULL;
 	dma_addr_t data_out_dma = 0;
@@ -742,7 +756,8 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg,
 	if (mpi_request->Function == MPI2_FUNCTION_SCSI_IO_REQUEST ||
 	    mpi_request->Function == MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH ||
 	    mpi_request->Function == MPI2_FUNCTION_SCSI_TASK_MGMT ||
-	    mpi_request->Function == MPI2_FUNCTION_SATA_PASSTHROUGH) {
+	    mpi_request->Function == MPI2_FUNCTION_SATA_PASSTHROUGH ||
+	    mpi_request->Function == MPI2_FUNCTION_NVME_ENCAPSULATED) {
 
 		device_handle = le16_to_cpu(mpi_request->FunctionDependent1);
 		if (!device_handle || (device_handle >
@@ -793,6 +808,38 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg,
 
 	init_completion(&ioc->ctl_cmds.done);
 	switch (mpi_request->Function) {
+	case MPI2_FUNCTION_NVME_ENCAPSULATED:
+	{
+		nvme_encap_request = (Mpi26NVMeEncapsulatedRequest_t *)request;
+		/*
+		 * Get the Physical Address of the sense buffer.
+		 * Use Error Response buffer address field to hold the sense
+		 * buffer address.
+		 * Clear the internal sense buffer, which will potentially hold
+		 * the Completion Queue Entry on return, or 0 if no Entry.
+		 * Build the PRPs and set direction bits.
+		 * Send the request.
+		 */
+		nvme_encap_request->ErrorResponseBaseAddress = ioc->sense_dma &
+		    0xFFFFFFFF00000000;
+		nvme_encap_request->ErrorResponseBaseAddress |=
+		    (U64)mpt3sas_base_get_sense_buffer_dma(ioc, smid);
+		nvme_encap_request->ErrorResponseAllocationLength =
+						NVME_ERROR_RESPONSE_SIZE;
+		memset(ioc->ctl_cmds.sense, 0, NVME_ERROR_RESPONSE_SIZE);
+		ioc->build_nvme_prp(ioc, smid, nvme_encap_request,
+		    data_out_dma, data_out_sz, data_in_dma, data_in_sz);
+		if (test_bit(device_handle, ioc->device_remove_in_progress)) {
+			dtmprintk(ioc, pr_info(MPT3SAS_FMT "handle(0x%04x) :"
+			    "ioctl failed due to device removal in progress\n",
+			    ioc->name, device_handle));
+			mpt3sas_base_free_smid(ioc, smid);
+			ret = -EINVAL;
+			goto out;
+		}
+		ioc->put_smid_nvme_encap(ioc, smid);
+		break;
+	}
 	case MPI2_FUNCTION_SCSI_IO_REQUEST:
 	case MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH:
 	{
@@ -1008,15 +1055,25 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg,
 		}
 	}
 
-	/* copy out sense to user */
+	/* copy out sense/NVMe Error Response to user */
 	if (karg.max_sense_bytes && (mpi_request->Function ==
 	    MPI2_FUNCTION_SCSI_IO_REQUEST || mpi_request->Function ==
-	    MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH)) {
-		sz = min_t(u32, karg.max_sense_bytes, SCSI_SENSE_BUFFERSIZE);
+	    MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH || mpi_request->Function ==
+	    MPI2_FUNCTION_NVME_ENCAPSULATED)) {
+		if (karg.sense_data_ptr == NULL) {
+			pr_info(MPT3SAS_FMT "Response buffer provided"
+			    " by application is NULL; Response data will"
+			    " not be returned.\n", ioc->name);
+			goto out;
+		}
+		sz_arg = (mpi_request->Function ==
+		MPI2_FUNCTION_NVME_ENCAPSULATED) ? NVME_ERROR_RESPONSE_SIZE :
+							SCSI_SENSE_BUFFERSIZE;
+		sz = min_t(u32, karg.max_sense_bytes, sz_arg);
 		if (copy_to_user(karg.sense_data_ptr, ioc->ctl_cmds.sense,
 		    sz)) {
 			pr_err("failure at %s:%d/%s()!\n", __FILE__,
-			    __LINE__, __func__);
+				__LINE__, __func__);
 			ret = -ENODATA;
 			goto out;
 		}
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:28 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:28 +0530
Subject: [PATCH 02/13] mpt3sas: SGL to PRP Translation for I/Os to NVMe devices
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-3-git-send-email-suganath-prabu.subramani@broadcom.com>

* Added support for translating the SGLs associated with incoming
commands either to IEE SGL or NVMe PRPs for NVMe devices.

* The hardware translation of IEEE SGL to NVMe PRPs has limitation
and if a command cannot be translated by hardware then it will go
to firmware and the firmware needs to translate it. And this will
have a performance reduction. To avoid that driver proactively
checks whether the translation will be done in hardware or not,
if not then driver try to translate inside the driver.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c      | 339 ++++++++++++++++++++++++++++++-
 drivers/scsi/mpt3sas/mpt3sas_base.h      |  41 +++-
 drivers/scsi/mpt3sas/mpt3sas_ctl.c       |   1 +
 drivers/scsi/mpt3sas/mpt3sas_scsih.c     |  14 +-
 drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |   2 +-
 5 files changed, 380 insertions(+), 17 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 11c6afe..1ad3cbb 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -59,6 +59,7 @@
 #include <linux/time.h>
 #include <linux/ktime.h>
 #include <linux/kthread.h>
+#include <asm/page.h>        /* To get host page size per arch */
 #include <linux/aer.h>
 
 
@@ -1344,7 +1345,218 @@ _base_build_sg(struct MPT3SAS_ADAPTER *ioc, void *psge,
 	}
 }
 
-/* IEEE format sgls */
+/**
+ * base_make_prp_nvme -
+ * Prepare PRPs(Physical Region Page)- SGLs specific to NVMe drives only
+ *
+ * @ioc:		per adapter object
+ * @scmd:		SCSI command from the mid-layer
+ * @mpi_request:	mpi request
+ * @smid:		msg Index
+ * @sge_count:		scatter gather element count.
+ *
+ * Returns:		true: PRPs are built
+ *			false: IEEE SGLs needs to be built
+ */
+void
+base_make_prp_nvme(struct MPT3SAS_ADAPTER *ioc,
+		struct scsi_cmnd *scmd,
+		Mpi25SCSIIORequest_t *mpi_request,
+		u16 smid, int sge_count)
+{
+	int sge_len, offset, num_prp_in_chain = 0;
+	Mpi25IeeeSgeChain64_t *main_chain_element, *ptr_first_sgl;
+	u64 *curr_buff;
+	dma_addr_t msg_phys;
+	u64 sge_addr;
+	u32 page_mask, page_mask_result;
+	struct scatterlist *sg_scmd;
+	u32 first_prp_len;
+	int data_len = scsi_bufflen(scmd);
+	u32 nvme_pg_size;
+
+	nvme_pg_size = max_t(u32, ioc->page_size, NVME_PRP_PAGE_SIZE);
+	/*
+	 * Nvme has a very convoluted prp format.  One prp is required
+	 * for each page or partial page. Driver need to split up OS sg_list
+	 * entries if it is longer than one page or cross a page
+	 * boundary.  Driver also have to insert a PRP list pointer entry as
+	 * the last entry in each physical page of the PRP list.
+	 *
+	 * NOTE: The first PRP "entry" is actually placed in the first
+	 * SGL entry in the main message as IEEE 64 format.  The 2nd
+	 * entry in the main message is the chain element, and the rest
+	 * of the PRP entries are built in the contiguous pcie buffer.
+	 */
+	page_mask = nvme_pg_size - 1;
+
+	/*
+	 * Native SGL is needed.
+	 * Put a chain element in main message frame that points to the first
+	 * chain buffer.
+	 *
+	 * NOTE:  The ChainOffset field must be 0 when using a chain pointer to
+	 *        a native SGL.
+	 */
+
+	/* Set main message chain element pointer */
+	main_chain_element = (pMpi25IeeeSgeChain64_t)&mpi_request->SGL;
+	/*
+	 * For NVMe the chain element needs to be the 2nd SG entry in the main
+	 * message.
+	 */
+	main_chain_element = (Mpi25IeeeSgeChain64_t *)
+		((u8 *)main_chain_element + sizeof(MPI25_IEEE_SGE_CHAIN64));
+
+	/*
+	 * For the PRP entries, use the specially allocated buffer of
+	 * contiguous memory.  Normal chain buffers can't be used
+	 * because each chain buffer would need to be the size of an OS
+	 * page (4k).
+	 */
+	curr_buff = mpt3sas_base_get_pcie_sgl(ioc, smid);
+	msg_phys = (dma_addr_t)mpt3sas_base_get_pcie_sgl_dma(ioc, smid);
+
+	main_chain_element->Address = cpu_to_le64(msg_phys);
+	main_chain_element->NextChainOffset = 0;
+	main_chain_element->Flags = MPI2_IEEE_SGE_FLAGS_CHAIN_ELEMENT |
+			MPI2_IEEE_SGE_FLAGS_SYSTEM_ADDR |
+			MPI26_IEEE_SGE_FLAGS_NSF_NVME_PRP;
+
+	/* Build first prp, sge need not to be page aligned*/
+	ptr_first_sgl = (pMpi25IeeeSgeChain64_t)&mpi_request->SGL;
+	sg_scmd = scsi_sglist(scmd);
+	sge_addr = sg_dma_address(sg_scmd);
+	sge_len = sg_dma_len(sg_scmd);
+
+	offset = (u32)(sge_addr & page_mask);
+	first_prp_len = nvme_pg_size - offset;
+
+	ptr_first_sgl->Address = cpu_to_le64(sge_addr);
+	ptr_first_sgl->Length = cpu_to_le32(first_prp_len);
+
+	data_len -= first_prp_len;
+
+	if (sge_len > first_prp_len) {
+		sge_addr += first_prp_len;
+		sge_len -= first_prp_len;
+	} else if (data_len && (sge_len == first_prp_len)) {
+		sg_scmd = sg_next(sg_scmd);
+		sge_addr = sg_dma_address(sg_scmd);
+		sge_len = sg_dma_len(sg_scmd);
+	}
+
+	for (;;) {
+		offset = (u32)(sge_addr & page_mask);
+
+		/* Put PRP pointer due to page boundary*/
+		page_mask_result = (uintptr_t)(curr_buff + 1) & page_mask;
+		if (unlikely(!page_mask_result)) {
+			scmd_printk(KERN_NOTICE,
+				scmd, "page boundary curr_buff: 0x%p\n",
+				curr_buff);
+			msg_phys += 8;
+			*curr_buff = cpu_to_le64(msg_phys);
+			curr_buff++;
+			num_prp_in_chain++;
+		}
+
+		*curr_buff = cpu_to_le64(sge_addr);
+		curr_buff++;
+		msg_phys += 8;
+		num_prp_in_chain++;
+
+		sge_addr += nvme_pg_size;
+		sge_len -= nvme_pg_size;
+		data_len -= nvme_pg_size;
+
+		if (data_len <= 0)
+			break;
+
+		if (sge_len > 0)
+			continue;
+
+		sg_scmd = sg_next(sg_scmd);
+		sge_addr = sg_dma_address(sg_scmd);
+		sge_len = sg_dma_len(sg_scmd);
+	}
+
+	main_chain_element->Length =
+		cpu_to_le32(num_prp_in_chain * sizeof(u64));
+	return;
+}
+
+static bool
+base_is_prp_possible(struct MPT3SAS_ADAPTER *ioc,
+	struct _pcie_device *pcie_device, struct scsi_cmnd *scmd, int sge_count)
+{
+	u32 data_length = 0;
+	struct scatterlist *sg_scmd;
+	bool build_prp = true;
+
+	data_length = cpu_to_le32(scsi_bufflen(scmd));
+	sg_scmd = scsi_sglist(scmd);
+
+	/* If Datalenth is <= 16K and number of SGE???s entries are <= 2
+	 * we built IEEE SGL
+	 */
+	if ((data_length <= NVME_PRP_PAGE_SIZE*4) && (sge_count <= 2))
+		build_prp = false;
+
+	return build_prp;
+}
+
+/**
+ * _base_check_pcie_native_sgl - This function is called for PCIe end devices to
+ * determine if the driver needs to build a native SGL.  If so, that native
+ * SGL is built in the special contiguous buffers allocated especially for
+ * PCIe SGL creation.  If the driver will not build a native SGL, return
+ * TRUE and a normal IEEE SGL will be built.  Currently this routine
+ * supports NVMe.
+ * @ioc: per adapter object
+ * @mpi_request: mf request pointer
+ * @smid: system request message index
+ * @scmd: scsi command
+ * @pcie_device: points to the PCIe device's info
+ *
+ * Returns 0 if native SGL was built, 1 if no SGL was built
+ */
+static int
+_base_check_pcie_native_sgl(struct MPT3SAS_ADAPTER *ioc,
+	Mpi25SCSIIORequest_t *mpi_request, u16 smid, struct scsi_cmnd *scmd,
+	struct _pcie_device *pcie_device)
+{
+	struct scatterlist *sg_scmd;
+	int sges_left;
+
+	/* Get the SG list pointer and info. */
+	sg_scmd = scsi_sglist(scmd);
+	sges_left = scsi_dma_map(scmd);
+	if (sges_left < 0) {
+		sdev_printk(KERN_ERR, scmd->device,
+			"scsi_dma_map failed: request for %d bytes!\n",
+			scsi_bufflen(scmd));
+		return 1;
+	}
+
+	/* Check if we need to build a native SG list. */
+	if (base_is_prp_possible(ioc, pcie_device,
+				scmd, sges_left) == 0) {
+		/* We built a native SG list, just return. */
+		goto out;
+	}
+
+	/*
+	 * Build native NVMe PRP.
+	 */
+	base_make_prp_nvme(ioc, scmd, mpi_request,
+			smid, sges_left);
+
+	return 0;
+out:
+	scsi_dma_unmap(scmd);
+	return 1;
+}
 
 /**
  * _base_add_sg_single_ieee - add sg element for IEEE format
@@ -1391,9 +1603,11 @@ _base_build_zero_len_sge_ieee(struct MPT3SAS_ADAPTER *ioc, void *paddr)
 
 /**
  * _base_build_sg_scmd - main sg creation routine
+ *		pcie_device is unused here!
  * @ioc: per adapter object
  * @scmd: scsi command
  * @smid: system request message index
+ * @unused: unused pcie_device pointer
  * Context: none.
  *
  * The main routine that builds scatter gather table from a given
@@ -1403,7 +1617,7 @@ _base_build_zero_len_sge_ieee(struct MPT3SAS_ADAPTER *ioc, void *paddr)
  */
 static int
 _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc,
-		struct scsi_cmnd *scmd, u16 smid)
+	struct scsi_cmnd *scmd, u16 smid, struct _pcie_device *unused)
 {
 	Mpi2SCSIIORequest_t *mpi_request;
 	dma_addr_t chain_dma;
@@ -1537,6 +1751,8 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc,
  * @ioc: per adapter object
  * @scmd: scsi command
  * @smid: system request message index
+ * @pcie_device: Pointer to pcie_device. If set, the pcie native sgl will be
+ * constructed on need.
  * Context: none.
  *
  * The main routine that builds scatter gather table from a given
@@ -1546,9 +1762,9 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc,
  */
 static int
 _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc,
-	struct scsi_cmnd *scmd, u16 smid)
+	struct scsi_cmnd *scmd, u16 smid, struct _pcie_device *pcie_device)
 {
-	Mpi2SCSIIORequest_t *mpi_request;
+	Mpi25SCSIIORequest_t *mpi_request;
 	dma_addr_t chain_dma;
 	struct scatterlist *sg_scmd;
 	void *sg_local, *chain;
@@ -1571,6 +1787,13 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc,
 	chain_sgl_flags = MPI2_IEEE_SGE_FLAGS_CHAIN_ELEMENT |
 	    MPI2_IEEE_SGE_FLAGS_SYSTEM_ADDR;
 
+	/* Check if we need to build a native SG list. */
+	if ((pcie_device) && (_base_check_pcie_native_sgl(ioc, mpi_request,
+			smid, scmd, pcie_device) == 0)) {
+		/* We built a native SG list, just return. */
+		return 0;
+	}
+
 	sg_scmd = scsi_sglist(scmd);
 	sges_left = scsi_dma_map(scmd);
 	if (sges_left < 0) {
@@ -1582,12 +1805,12 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc,
 
 	sg_local = &mpi_request->SGL;
 	sges_in_segment = (ioc->request_sz -
-	    offsetof(Mpi2SCSIIORequest_t, SGL))/ioc->sge_size_ieee;
+		   offsetof(Mpi25SCSIIORequest_t, SGL))/ioc->sge_size_ieee;
 	if (sges_left <= sges_in_segment)
 		goto fill_in_last_segment;
 
 	mpi_request->ChainOffset = (sges_in_segment - 1 /* chain element */) +
-	    (offsetof(Mpi2SCSIIORequest_t, SGL)/ioc->sge_size_ieee);
+	    (offsetof(Mpi25SCSIIORequest_t, SGL)/ioc->sge_size_ieee);
 
 	/* fill in main message segment when there is a chain following */
 	while (sges_in_segment > 1) {
@@ -2267,6 +2490,33 @@ mpt3sas_base_get_sense_buffer_dma(struct MPT3SAS_ADAPTER *ioc, u16 smid)
 }
 
 /**
+ * mpt3sas_base_get_pcie_sgl - obtain a PCIe SGL virt addr
+ * @ioc: per adapter object
+ * @smid: system request message index
+ *
+ * Returns virt pointer to a PCIe SGL.
+ */
+void *
+mpt3sas_base_get_pcie_sgl(struct MPT3SAS_ADAPTER *ioc, u16 smid)
+{
+	return (void *)(ioc->scsi_lookup[smid - 1].pcie_sg_list.pcie_sgl);
+}
+
+/**
+ * mpt3sas_base_get_pcie_sgl_dma - obtain a PCIe SGL dma addr
+ * @ioc: per adapter object
+ * @smid: system request message index
+ *
+ * Returns phys pointer to the address of the PCIe buffer.
+ */
+void *
+mpt3sas_base_get_pcie_sgl_dma(struct MPT3SAS_ADAPTER *ioc, u16 smid)
+{
+	return (void *)(uintptr_t)
+		(ioc->scsi_lookup[smid - 1].pcie_sg_list.pcie_sgl_dma);
+}
+
+/**
  * mpt3sas_base_get_reply_virt_addr - obtain reply frames virt address
  * @ioc: per adapter object
  * @phys_addr: lower 32 physical addr of the reply
@@ -2945,6 +3195,11 @@ _base_display_ioc_capabilities(struct MPT3SAS_ADAPTER *ioc)
 
 	_base_display_OEMs_branding(ioc);
 
+	if (ioc->facts.ProtocolFlags & MPI2_IOCFACTS_PROTOCOL_NVME_DEVICES) {
+		pr_info("%sNVMe", i ? "," : "");
+		i++;
+	}
+
 	pr_info(MPT3SAS_FMT "Protocol=(", ioc->name);
 
 	if (ioc->facts.ProtocolFlags & MPI2_IOCFACTS_PROTOCOL_SCSI_INITIATOR) {
@@ -3245,6 +3500,17 @@ _base_release_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 		kfree(ioc->reply_post);
 	}
 
+	if (ioc->pcie_sgl_dma_pool) {
+		for (i = 0; i < ioc->scsiio_depth; i++) {
+			if (ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl)
+				pci_pool_free(ioc->pcie_sgl_dma_pool,
+				ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl,
+				ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl_dma);
+		}
+		if (ioc->pcie_sgl_dma_pool)
+			pci_pool_destroy(ioc->pcie_sgl_dma_pool);
+	}
+
 	if (ioc->config_page) {
 		dexitprintk(ioc, pr_info(MPT3SAS_FMT
 		    "config_page(0x%p): free\n", ioc->name,
@@ -3286,7 +3552,7 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 	u16 chains_needed_per_io;
 	u32 sz, total_sz, reply_post_free_sz;
 	u32 retry_sz;
-	u16 max_request_credit;
+	u16 max_request_credit, nvme_blocks_needed;
 	unsigned short sg_tablesize;
 	u16 sge_size;
 	int i;
@@ -3630,7 +3896,52 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 		"internal(0x%p): depth(%d), start smid(%d)\n",
 		ioc->name, ioc->internal,
 	    ioc->internal_depth, ioc->internal_smid));
+	/*
+	 * The number of NVMe page sized blocks needed is:
+	 *     (((sg_tablesize * 8) - 1) / (page_size - 8)) + 1
+	 * ((sg_tablesize * 8) - 1) is the max PRP's minus the first PRP entry
+	 * that is placed in the main message frame.  8 is the size of each PRP
+	 * entry or PRP list pointer entry.  8 is subtracted from page_size
+	 * because of the PRP list pointer entry at the end of a page, so this
+	 * is not counted as a PRP entry.  The 1 added page is a round up.
+	 *
+	 * To avoid allocation failures due to the amount of memory that could
+	 * be required for NVMe PRP's, only each set of NVMe blocks will be
+	 * contiguous, so a new set is allocated for each possible I/O.
+	 */
+	if (ioc->facts.ProtocolFlags & MPI2_IOCFACTS_PROTOCOL_NVME_DEVICES) {
+		nvme_blocks_needed =
+			(ioc->shost->sg_tablesize * NVME_PRP_SIZE) - 1;
+		nvme_blocks_needed /= (ioc->page_size - NVME_PRP_SIZE);
+		nvme_blocks_needed++;
+
+		sz = nvme_blocks_needed * ioc->page_size;
+		ioc->pcie_sgl_dma_pool =
+			pci_pool_create("PCIe SGL pool", ioc->pdev, sz, 16, 0);
+		if (!ioc->pcie_sgl_dma_pool) {
+			pr_info(MPT3SAS_FMT
+			    "PCIe SGL pool: pci_pool_create failed\n",
+			    ioc->name);
+			goto out;
+		}
+		for (i = 0; i < ioc->scsiio_depth; i++) {
+			ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl =
+					pci_pool_alloc(ioc->pcie_sgl_dma_pool,
+					GFP_KERNEL,
+				&ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl_dma);
+			if (!ioc->scsi_lookup[i].pcie_sg_list.pcie_sgl) {
+				pr_info(MPT3SAS_FMT
+				    "PCIe SGL pool: pci_pool_alloc failed\n",
+				    ioc->name);
+				goto out;
+			}
+		}
 
+		dinitprintk(ioc, pr_info(MPT3SAS_FMT "PCIe sgl pool depth(%d), "
+			"element_size(%d), pool_size(%d kB)\n", ioc->name,
+			ioc->scsiio_depth, sz, (sz * ioc->scsiio_depth)/1024));
+		total_sz += sz * ioc->scsiio_depth;
+	}
 	/* sense buffers, 4 byte align */
 	sz = ioc->scsiio_depth * SCSI_SENSE_BUFFERSIZE;
 	ioc->sense_dma_pool = dma_pool_create("sense pool", &ioc->pdev->dev, sz,
@@ -4475,6 +4786,19 @@ _base_get_ioc_facts(struct MPT3SAS_ADAPTER *ioc)
 	    le16_to_cpu(mpi_reply.HighPriorityCredit);
 	facts->ReplyFrameSize = mpi_reply.ReplyFrameSize;
 	facts->MaxDevHandle = le16_to_cpu(mpi_reply.MaxDevHandle);
+	facts->CurrentHostPageSize = mpi_reply.CurrentHostPageSize;
+
+	/*
+	 * Get the Page Size from IOC Facts. If it's 0, default to 4k.
+	 */
+	ioc->page_size = 1 << facts->CurrentHostPageSize;
+	if (ioc->page_size == 1) {
+		pr_info(MPT3SAS_FMT "CurrentHostPageSize is 0: Setting "
+			"default host page size to 4k\n", ioc->name);
+		ioc->page_size = 1 << MPT3SAS_HOST_PAGE_SIZE_4K;
+	}
+	dinitprintk(ioc, pr_info(MPT3SAS_FMT "CurrentHostPageSize(%d)\n",
+		ioc->name, facts->CurrentHostPageSize));
 
 	dinitprintk(ioc, pr_info(MPT3SAS_FMT
 		"hba queue depth(%d), max chains per io(%d)\n",
@@ -4514,6 +4838,7 @@ _base_send_ioc_init(struct MPT3SAS_ADAPTER *ioc)
 	mpi_request.VP_ID = 0;
 	mpi_request.MsgVersion = cpu_to_le16(ioc->hba_mpi_version_belonged);
 	mpi_request.HeaderVersion = cpu_to_le16(MPI2_HEADER_VERSION);
+	mpi_request.HostPageSize = MPT3SAS_HOST_PAGE_SIZE_4K;
 
 	if (_base_is_controller_msix_enabled(ioc))
 		mpi_request.HostMSIxVectors = ioc->reply_queue_count;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index dd04a28..4758729 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -54,6 +54,7 @@
 #include "mpi/mpi2_raid.h"
 #include "mpi/mpi2_tool.h"
 #include "mpi/mpi2_sas.h"
+#include "mpi/mpi2_pci.h"
 
 #include <scsi/scsi.h>
 #include <scsi/scsi_cmnd.h>
@@ -115,6 +116,7 @@
 #define MPT3SAS_KDUMP_SCSI_IO_DEPTH	200
 
 #define MPT3SAS_RAID_MAX_SECTORS	8192
+#define MPT3SAS_HOST_PAGE_SIZE_4K	12
 
 #define MPT_NAME_LENGTH			32	/* generic length of strings */
 #define MPT_STRING_LENGTH		64
@@ -133,6 +135,15 @@
 #define DEFAULT_NUM_FWCHAIN_ELEMTS	8
 
 /*
+ * NVMe defines
+ */
+#define	NVME_PRP_SIZE			8	/* PRP size */
+#define	NVME_CMD_PRP1_OFFSET		24	/* PRP1 offset in NVMe cmd */
+#define	NVME_CMD_PRP2_OFFSET		32	/* PRP2 offset in NVMe cmd */
+#define	NVME_ERROR_RESPONSE_SIZE	16	/* Max NVME Error Response */
+#define	NVME_PRP_PAGE_SIZE		4096	/* Page size */
+
+/*
  * reset phases
  */
 #define MPT3_IOC_PRE_RESET		1 /* prior to host reset */
@@ -737,6 +748,16 @@ enum reset_type {
 };
 
 /**
+ * struct pcie_sg_list - PCIe SGL buffer (contiguous per I/O)
+ * @pcie_sgl: PCIe native SGL for NVMe devices
+ * @pcie_sgl_dma: physical address
+ */
+struct pcie_sg_list {
+	void            *pcie_sgl;
+	dma_addr_t      pcie_sgl_dma;
+};
+
+/**
  * struct chain_tracker - firmware chain tracker
  * @chain_buffer: chain buffer
  * @chain_buffer_dma: physical address
@@ -762,6 +783,7 @@ struct scsiio_tracker {
 	struct scsi_cmnd *scmd;
 	u8	cb_idx;
 	u8	direct_io;
+	struct pcie_sg_list pcie_sg_list;
 	struct list_head chain_list;
 	struct list_head tracker_list;
 	u16     msix_io;
@@ -835,13 +857,19 @@ typedef void (*MPT_ADD_SGE)(void *paddr, u32 flags_length, dma_addr_t dma_addr);
 
 /* SAS3.0 support */
 typedef int (*MPT_BUILD_SG_SCMD)(struct MPT3SAS_ADAPTER *ioc,
-		struct scsi_cmnd *scmd, u16 smid);
+	struct scsi_cmnd *scmd, u16 smid, struct _pcie_device *pcie_device);
 typedef void (*MPT_BUILD_SG)(struct MPT3SAS_ADAPTER *ioc, void *psge,
 		dma_addr_t data_out_dma, size_t data_out_sz,
 		dma_addr_t data_in_dma, size_t data_in_sz);
 typedef void (*MPT_BUILD_ZERO_LEN_SGE)(struct MPT3SAS_ADAPTER *ioc,
 		void *paddr);
 
+/* SAS3.5 support */
+typedef void (*NVME_BUILD_PRP)(struct MPT3SAS_ADAPTER *ioc, u16 smid,
+	Mpi26NVMeEncapsulatedRequest_t *nvme_encap_request,
+	dma_addr_t data_out_dma, size_t data_out_sz, dma_addr_t data_in_dma,
+	size_t data_in_sz);
+
 /* To support atomic and non atomic descriptors*/
 typedef void (*PUT_SMID_IO_FP_HIP) (struct MPT3SAS_ADAPTER *ioc, u16 smid,
 	u16 funcdep);
@@ -884,6 +912,7 @@ struct mpt3sas_facts {
 	u16			MaxDevHandle;
 	u16			MaxPersistentEntries;
 	u16			MinDevHandle;
+	u8			CurrentHostPageSize;
 };
 
 struct mpt3sas_port_facts {
@@ -1223,6 +1252,11 @@ struct MPT3SAS_ADAPTER {
 	int		pending_io_count;
 	wait_queue_head_t reset_wq;
 
+	/* PCIe SGL */
+	struct dma_pool *pcie_sgl_dma_pool;
+	/* Host Page Size */
+	u32		page_size;
+
 	/* chain */
 	struct chain_tracker *chain_lookup;
 	struct list_head free_chain_list;
@@ -1356,7 +1390,8 @@ void *mpt3sas_base_get_msg_frame(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 void *mpt3sas_base_get_sense_buffer(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 __le32 mpt3sas_base_get_sense_buffer_dma(struct MPT3SAS_ADAPTER *ioc,
 	u16 smid);
-
+void *mpt3sas_base_get_pcie_sgl(struct MPT3SAS_ADAPTER *ioc, u16 smid);
+void *mpt3sas_base_get_pcie_sgl_dma(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 void mpt3sas_base_sync_reply_irqs(struct MPT3SAS_ADAPTER *ioc);
 
 /* hi-priority queue */
@@ -1570,7 +1605,7 @@ void
 mpt3sas_scsi_direct_io_set(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 direct_io);
 void
 mpt3sas_setup_direct_io(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
-	struct _raid_device *raid_device, Mpi2SCSIIORequest_t *mpi_request,
+	struct _raid_device *raid_device, Mpi25SCSIIORequest_t *mpi_request,
 	u16 smid);
 
 /* NCQ Prio Handling Check */
diff --git a/drivers/scsi/mpt3sas/mpt3sas_ctl.c b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
index d448fed..67c7280 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_ctl.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
@@ -299,6 +299,7 @@ mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index,
 			}
 		}
 	}
+
 	_ctl_display_some_debug(ioc, smid, "ctl_done", mpi_reply);
 	ioc->ctl_cmds.status &= ~MPT3_CMD_PENDING;
 	complete(&ioc->ctl_cmds.done);
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 4c9b63e..51dd92c 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4256,7 +4256,7 @@ _scsih_flush_running_cmds(struct MPT3SAS_ADAPTER *ioc)
  */
 static void
 _scsih_setup_eedp(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
-	Mpi2SCSIIORequest_t *mpi_request)
+	Mpi25SCSIIORequest_t *mpi_request)
 {
 	u16 eedp_flags;
 	unsigned char prot_op = scsi_get_prot_op(scmd);
@@ -4359,7 +4359,8 @@ scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd)
 	struct _raid_device *raid_device;
 	struct request *rq = scmd->request;
 	int class;
-	Mpi2SCSIIORequest_t *mpi_request;
+	Mpi25SCSIIORequest_t *mpi_request;
+	struct _pcie_device *pcie_device = NULL;
 	u32 mpi_control;
 	u16 smid;
 	u16 handle;
@@ -4447,7 +4448,7 @@ scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd)
 		goto out;
 	}
 	mpi_request = mpt3sas_base_get_msg_frame(ioc, smid);
-	memset(mpi_request, 0, sizeof(Mpi2SCSIIORequest_t));
+	memset(mpi_request, 0, ioc->request_sz);
 	_scsih_setup_eedp(ioc, scmd, mpi_request);
 
 	if (scmd->cmd_len == 32)
@@ -4466,13 +4467,14 @@ scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd)
 	mpi_request->SenseBufferLength = SCSI_SENSE_BUFFERSIZE;
 	mpi_request->SenseBufferLowAddress =
 	    mpt3sas_base_get_sense_buffer_dma(ioc, smid);
-	mpi_request->SGLOffset0 = offsetof(Mpi2SCSIIORequest_t, SGL) / 4;
+	mpi_request->SGLOffset0 = offsetof(Mpi25SCSIIORequest_t, SGL) / 4;
 	int_to_scsilun(sas_device_priv_data->lun, (struct scsi_lun *)
 	    mpi_request->LUN);
 	memcpy(mpi_request->CDB.CDB32, scmd->cmnd, scmd->cmd_len);
 
 	if (mpi_request->DataLength) {
-		if (ioc->build_sg_scmd(ioc, scmd, smid)) {
+		pcie_device = sas_target_priv_data->pcie_dev;
+		if (ioc->build_sg_scmd(ioc, scmd, smid, pcie_device)) {
 			mpt3sas_base_free_smid(ioc, smid);
 			goto out;
 		}
@@ -4925,7 +4927,7 @@ out_unlock:
 static u8
 _scsih_io_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index, u32 reply)
 {
-	Mpi2SCSIIORequest_t *mpi_request;
+	Mpi25SCSIIORequest_t *mpi_request;
 	Mpi2SCSIIOReply_t *mpi_reply;
 	struct scsi_cmnd *scmd;
 	u16 ioc_status;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_warpdrive.c b/drivers/scsi/mpt3sas/mpt3sas_warpdrive.c
index 540bd50..ced7d9f 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_warpdrive.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_warpdrive.c
@@ -299,7 +299,7 @@ mpt3sas_scsi_direct_io_set(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 direct_io)
  */
 void
 mpt3sas_setup_direct_io(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
-	struct _raid_device *raid_device, Mpi2SCSIIORequest_t *mpi_request,
+	struct _raid_device *raid_device, Mpi25SCSIIORequest_t *mpi_request,
 	u16 smid)
 {
 	sector_t v_lba, p_lba, stripe_off, column, io_size;
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:31 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:31 +0530
Subject: [PATCH 05/13] mpt3sas: API's to remove nvme drive from sml
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-6-git-send-email-suganath-prabu.subramani@broadcom.com>

Below API's are included in nvme drive remove path.
_scsih_pcie_device_remove_by_handle
_scsih_pcie_device_remove_from_sml

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 148 ++++++++++++++++++++++++++++++++++-
 1 file changed, 145 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index d3f8a10..d83560c 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -73,6 +73,8 @@ static void _scsih_remove_device(struct MPT3SAS_ADAPTER *ioc,
 static int _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	u8 retry_count, u8 is_pd);
 static int _scsih_pcie_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle);
+static void _scsih_pcie_device_remove_from_sml(struct MPT3SAS_ADAPTER *ioc,
+	struct _pcie_device *pcie_device);
 static u8 _scsih_check_for_pending_tm(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 
 /* global parameters */
@@ -1126,6 +1128,41 @@ _scsih_pcie_device_remove(struct MPT3SAS_ADAPTER *ioc,
 		pcie_device_put(pcie_device);
 	}
 }
+
+
+/**
+ * _scsih_pcie_device_remove_by_handle - removing pcie device object by handle
+ * @ioc: per adapter object
+ * @handle: device handle
+ *
+ * Return nothing.
+ */
+static void
+_scsih_pcie_device_remove_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
+{
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+	int was_on_pcie_device_list = 0;
+
+	if (ioc->shost_recovery)
+		return;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device = __mpt3sas_get_pdev_by_handle(ioc, handle);
+	if (pcie_device) {
+		if (!list_empty(&pcie_device->list)) {
+			list_del_init(&pcie_device->list);
+			was_on_pcie_device_list = 1;
+			pcie_device_put(pcie_device);
+		}
+	}
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+	if (was_on_pcie_device_list) {
+		_scsih_pcie_device_remove_from_sml(ioc, pcie_device);
+		pcie_device_put(pcie_device);
+	}
+}
+
 /**
  * _scsih_pcie_device_add - add pcie_device object
  * @ioc: per adapter object
@@ -6567,6 +6604,83 @@ _scsih_check_pcie_access_status(struct MPT3SAS_ADAPTER *ioc, u64 wwid,
 			(unsigned long long)wwid, handle);
 	return rc;
 }
+
+/**
+ * _scsih_pcie_device_remove_from_sml -  removing pcie device
+ * from SML and free up associated memory
+ * @ioc: per adapter object
+ * @pcie_device: the pcie_device object
+ *
+ * Return nothing.
+ */
+static void
+_scsih_pcie_device_remove_from_sml(struct MPT3SAS_ADAPTER *ioc,
+	struct _pcie_device *pcie_device)
+{
+	struct MPT3SAS_TARGET *sas_target_priv_data;
+
+	dewtprintk(ioc, pr_info(MPT3SAS_FMT
+	    "%s: enter: handle(0x%04x), wwid(0x%016llx)\n", ioc->name, __func__,
+	    pcie_device->handle, (unsigned long long)
+	    pcie_device->wwid));
+	if (pcie_device->enclosure_handle != 0)
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+		    "%s: enter: enclosure logical id(0x%016llx), slot(%d)\n",
+		    ioc->name, __func__,
+		    (unsigned long long)pcie_device->enclosure_logical_id,
+		    pcie_device->slot));
+	if (pcie_device->connector_name[0] != '\0')
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+		    "%s: enter: enclosure level(0x%04x), connector name( %s)\n",
+		    ioc->name, __func__,
+		    pcie_device->enclosure_level,
+		    pcie_device->connector_name));
+
+	if (pcie_device->starget && pcie_device->starget->hostdata) {
+		sas_target_priv_data = pcie_device->starget->hostdata;
+		sas_target_priv_data->deleted = 1;
+		_scsih_ublock_io_device(ioc, pcie_device->wwid);
+		sas_target_priv_data->handle = MPT3SAS_INVALID_DEVICE_HANDLE;
+	}
+
+	pr_info(MPT3SAS_FMT
+		"removing handle(0x%04x), wwid (0x%016llx)\n",
+		ioc->name, pcie_device->handle,
+		(unsigned long long) pcie_device->wwid);
+	if (pcie_device->enclosure_handle != 0)
+		pr_info(MPT3SAS_FMT
+		    "removing : enclosure logical id(0x%016llx), slot(%d)\n",
+		    ioc->name,
+		    (unsigned long long)pcie_device->enclosure_logical_id,
+		    pcie_device->slot);
+	if (pcie_device->connector_name[0] != '\0')
+		pr_info(MPT3SAS_FMT
+		    "removing: enclosure level(0x%04x), connector name( %s)\n",
+		    ioc->name, pcie_device->enclosure_level,
+		    pcie_device->connector_name);
+
+	if (pcie_device->starget)
+		scsi_remove_target(&pcie_device->starget->dev);
+	dewtprintk(ioc, pr_info(MPT3SAS_FMT
+	    "%s: exit: handle(0x%04x), wwid(0x%016llx)\n", ioc->name, __func__,
+	    pcie_device->handle, (unsigned long long)
+	    pcie_device->wwid));
+	if (pcie_device->enclosure_handle != 0)
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+			"%s: exit: enclosure logical id(0x%016llx), slot(%d)\n",
+			ioc->name, __func__,
+			(unsigned long long)pcie_device->enclosure_logical_id,
+			pcie_device->slot));
+	if (pcie_device->connector_name[0] != '\0')
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+		    "%s: exit: enclosure level(0x%04x), connector name( %s)\n",
+		    ioc->name, __func__, pcie_device->enclosure_level,
+		    pcie_device->connector_name));
+
+	kfree(pcie_device->serial_number);
+}
+
+
 /**
  * _scsih_pcie_check_device - checking device responsiveness
  * @ioc: per adapter object
@@ -8186,17 +8300,18 @@ _scsih_search_responding_expanders(struct MPT3SAS_ADAPTER *ioc)
 }
 
 /**
- * _scsih_remove_unresponding_sas_devices - removing unresponding devices
+ * _scsih_remove_unresponding_devices - removing unresponding devices
  * @ioc: per adapter object
  *
  * Return nothing.
  */
 static void
-_scsih_remove_unresponding_sas_devices(struct MPT3SAS_ADAPTER *ioc)
+_scsih_remove_unresponding_devices(struct MPT3SAS_ADAPTER *ioc)
 {
 	struct _sas_device *sas_device, *sas_device_next;
 	struct _sas_node *sas_expander, *sas_expander_next;
 	struct _raid_device *raid_device, *raid_device_next;
+	struct _pcie_device *pcie_device, *pcie_device_next;
 	struct list_head tmp_list;
 	unsigned long flags;
 	LIST_HEAD(head);
@@ -8230,6 +8345,26 @@ _scsih_remove_unresponding_sas_devices(struct MPT3SAS_ADAPTER *ioc)
 		sas_device_put(sas_device);
 	}
 
+	pr_info(MPT3SAS_FMT
+		" Removing unresponding devices: pcie end-devices\n"
+		, ioc->name);
+	INIT_LIST_HEAD(&head);
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	list_for_each_entry_safe(pcie_device, pcie_device_next,
+	    &ioc->pcie_device_list, list) {
+		if (!pcie_device->responding)
+			list_move_tail(&pcie_device->list, &head);
+		else
+			pcie_device->responding = 0;
+	}
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	list_for_each_entry_safe(pcie_device, pcie_device_next, &head, list) {
+		_scsih_pcie_device_remove_from_sml(ioc, pcie_device);
+		list_del_init(&pcie_device->list);
+		pcie_device_put(pcie_device);
+	}
+
 	/* removing unresponding volumes */
 	if (ioc->ir_firmware) {
 		pr_info(MPT3SAS_FMT "removing unresponding devices: volumes\n",
@@ -8631,7 +8766,7 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 				goto out;
 			ssleep(1);
 		}
-		_scsih_remove_unresponding_sas_devices(ioc);
+		_scsih_remove_unresponding_devices(ioc);
 		_scsih_scan_for_devices_after_reset(ioc);
 		break;
 	case MPT3SAS_PORT_ENABLE_COMPLETE:
@@ -9019,6 +9154,7 @@ static void scsih_remove(struct pci_dev *pdev)
 	struct _sas_port *mpt3sas_port, *next_port;
 	struct _raid_device *raid_device, *next;
 	struct MPT3SAS_TARGET *sas_target_priv_data;
+	struct _pcie_device *pcie_device, *pcienext;
 	struct workqueue_struct	*wq;
 	unsigned long flags;
 
@@ -9047,6 +9183,12 @@ static void scsih_remove(struct pci_dev *pdev)
 		    (unsigned long long) raid_device->wwid);
 		_scsih_raid_device_remove(ioc, raid_device);
 	}
+	list_for_each_entry_safe(pcie_device, pcienext, &ioc->pcie_device_list,
+		list) {
+		_scsih_pcie_device_remove_from_sml(ioc, pcie_device);
+		list_del_init(&pcie_device->list);
+		pcie_device_put(pcie_device);
+	}
 
 	/* free ports attached to the sas_host */
 	list_for_each_entry_safe(mpt3sas_port, next_port,
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:30 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:30 +0530
Subject: [PATCH 04/13] mpt3sas: API 's to support NVMe drive addition to  SML
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-5-git-send-email-suganath-prabu.subramani@broadcom.com>

Below Functions are added in various paths to support NVMe
drive addition.

_scsih_pcie_add_device
_scsih_pcie_device_add
_scsih_pcie_device_init_add
_scsih_check_pcie_access_status
_scsih_pcie_check_device

mpt3sas_get_pdev_by_handle

mpt3sas_config_get_pcie_device_pg0
mpt3sas_config_get_pcie_device_pg2

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.h   |  10 +
 drivers/scsi/mpt3sas/mpt3sas_config.c | 100 ++++++++
 drivers/scsi/mpt3sas/mpt3sas_scsih.c  | 457 +++++++++++++++++++++++++++++++++-
 3 files changed, 565 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 034b34d..a4ec459 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1464,6 +1464,10 @@ struct _sas_device *mpt3sas_get_sdev_by_addr(
 	 struct MPT3SAS_ADAPTER *ioc, u64 sas_address);
 struct _sas_device *__mpt3sas_get_sdev_by_addr(
 	 struct MPT3SAS_ADAPTER *ioc, u64 sas_address);
+struct _sas_device *mpt3sas_get_sdev_by_handle(struct MPT3SAS_ADAPTER *ioc,
+	u16 handle);
+struct _pcie_device *mpt3sas_get_pdev_by_handle(struct MPT3SAS_ADAPTER *ioc,
+	u16 handle);
 
 void mpt3sas_port_enable_complete(struct MPT3SAS_ADAPTER *ioc);
 struct _raid_device *
@@ -1502,6 +1506,12 @@ int mpt3sas_config_get_sas_device_pg0(struct MPT3SAS_ADAPTER *ioc,
 int mpt3sas_config_get_sas_device_pg1(struct MPT3SAS_ADAPTER *ioc,
 	Mpi2ConfigReply_t *mpi_reply, Mpi2SasDevicePage1_t *config_page,
 	u32 form, u32 handle);
+int mpt3sas_config_get_pcie_device_pg0(struct MPT3SAS_ADAPTER *ioc,
+	Mpi2ConfigReply_t *mpi_reply, Mpi26PCIeDevicePage0_t *config_page,
+	u32 form, u32 handle);
+int mpt3sas_config_get_pcie_device_pg2(struct MPT3SAS_ADAPTER *ioc,
+	Mpi2ConfigReply_t *mpi_reply, Mpi26PCIeDevicePage2_t *config_page,
+	u32 form, u32 handle);
 int mpt3sas_config_get_sas_iounit_pg0(struct MPT3SAS_ADAPTER *ioc,
 	Mpi2ConfigReply_t *mpi_reply, Mpi2SasIOUnitPage0_t *config_page,
 	u16 sz);
diff --git a/drivers/scsi/mpt3sas/mpt3sas_config.c b/drivers/scsi/mpt3sas/mpt3sas_config.c
index dd62701..1c747cf 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_config.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_config.c
@@ -150,6 +150,24 @@ _config_display_some_debug(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 		case MPI2_CONFIG_EXTPAGETYPE_DRIVER_MAPPING:
 			desc = "driver_mapping";
 			break;
+		case MPI2_CONFIG_EXTPAGETYPE_SAS_PORT:
+			desc = "sas_port";
+			break;
+		case MPI2_CONFIG_EXTPAGETYPE_EXT_MANUFACTURING:
+			desc = "ext_manufacturing";
+			break;
+		case MPI2_CONFIG_EXTPAGETYPE_PCIE_IO_UNIT:
+			desc = "pcie_io_unit";
+			break;
+		case MPI2_CONFIG_EXTPAGETYPE_PCIE_SWITCH:
+			desc = "pcie_switch";
+			break;
+		case MPI2_CONFIG_EXTPAGETYPE_PCIE_DEVICE:
+			desc = "pcie_device";
+			break;
+		case MPI2_CONFIG_EXTPAGETYPE_PCIE_LINK:
+			desc = "pcie_link";
+			break;
 		}
 		break;
 	}
@@ -1053,6 +1071,88 @@ mpt3sas_config_get_sas_device_pg1(struct MPT3SAS_ADAPTER *ioc,
 }
 
 /**
+ * mpt3sas_config_get_pcie_device_pg0 - obtain pcie device page 0
+ * @ioc: per adapter object
+ * @mpi_reply: reply mf payload returned from firmware
+ * @config_page: contents of the config page
+ * @form: GET_NEXT_HANDLE or HANDLE
+ * @handle: device handle
+ * Context: sleep.
+ *
+ * Returns 0 for success, non-zero for failure.
+ */
+int
+mpt3sas_config_get_pcie_device_pg0(struct MPT3SAS_ADAPTER *ioc,
+	Mpi2ConfigReply_t *mpi_reply, Mpi26PCIeDevicePage0_t *config_page,
+	u32 form, u32 handle)
+{
+	Mpi2ConfigRequest_t mpi_request;
+	int r;
+
+	memset(&mpi_request, 0, sizeof(Mpi2ConfigRequest_t));
+	mpi_request.Function = MPI2_FUNCTION_CONFIG;
+	mpi_request.Action = MPI2_CONFIG_ACTION_PAGE_HEADER;
+	mpi_request.Header.PageType = MPI2_CONFIG_PAGETYPE_EXTENDED;
+	mpi_request.ExtPageType = MPI2_CONFIG_EXTPAGETYPE_PCIE_DEVICE;
+	mpi_request.Header.PageVersion = MPI26_PCIEDEVICE0_PAGEVERSION;
+	mpi_request.Header.PageNumber = 0;
+	ioc->build_zero_len_sge_mpi(ioc, &mpi_request.PageBufferSGE);
+	r = _config_request(ioc, &mpi_request, mpi_reply,
+			MPT3_CONFIG_PAGE_DEFAULT_TIMEOUT, NULL, 0);
+	if (r)
+		goto out;
+
+	mpi_request.PageAddress = cpu_to_le32(form | handle);
+	mpi_request.Action = MPI2_CONFIG_ACTION_PAGE_READ_CURRENT;
+	r = _config_request(ioc, &mpi_request, mpi_reply,
+			MPT3_CONFIG_PAGE_DEFAULT_TIMEOUT, config_page,
+			sizeof(*config_page));
+out:
+	return r;
+}
+
+/**
+ * mpt3sas_config_get_pcie_device_pg2 - obtain pcie device page 2
+ * @ioc: per adapter object
+ * @mpi_reply: reply mf payload returned from firmware
+ * @config_page: contents of the config page
+ * @form: GET_NEXT_HANDLE or HANDLE
+ * @handle: device handle
+ * Context: sleep.
+ *
+ * Returns 0 for success, non-zero for failure.
+ */
+int
+mpt3sas_config_get_pcie_device_pg2(struct MPT3SAS_ADAPTER *ioc,
+	Mpi2ConfigReply_t *mpi_reply, Mpi26PCIeDevicePage2_t *config_page,
+	u32 form, u32 handle)
+{
+	Mpi2ConfigRequest_t mpi_request;
+	int r;
+
+	memset(&mpi_request, 0, sizeof(Mpi2ConfigRequest_t));
+	mpi_request.Function = MPI2_FUNCTION_CONFIG;
+	mpi_request.Action = MPI2_CONFIG_ACTION_PAGE_HEADER;
+	mpi_request.Header.PageType = MPI2_CONFIG_PAGETYPE_EXTENDED;
+	mpi_request.ExtPageType = MPI2_CONFIG_EXTPAGETYPE_PCIE_DEVICE;
+	mpi_request.Header.PageVersion = MPI26_PCIEDEVICE2_PAGEVERSION;
+	mpi_request.Header.PageNumber = 2;
+	ioc->build_zero_len_sge_mpi(ioc, &mpi_request.PageBufferSGE);
+	r = _config_request(ioc, &mpi_request, mpi_reply,
+			MPT3_CONFIG_PAGE_DEFAULT_TIMEOUT, NULL, 0);
+	if (r)
+		goto out;
+
+	mpi_request.PageAddress = cpu_to_le32(form | handle);
+	mpi_request.Action = MPI2_CONFIG_ACTION_PAGE_READ_CURRENT;
+	r = _config_request(ioc, &mpi_request, mpi_reply,
+			MPT3_CONFIG_PAGE_DEFAULT_TIMEOUT, config_page,
+			sizeof(*config_page));
+out:
+	return r;
+}
+
+/**
  * mpt3sas_config_get_number_hba_phys - obtain number of phys on the host
  * @ioc: per adapter object
  * @num_phys: pointer returned with the number of phys
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 51dd92c..d3f8a10 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -72,7 +72,7 @@ static void _scsih_remove_device(struct MPT3SAS_ADAPTER *ioc,
 	struct _sas_device *sas_device);
 static int _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	u8 retry_count, u8 is_pd);
-
+static int _scsih_pcie_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle);
 static u8 _scsih_check_for_pending_tm(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 
 /* global parameters */
@@ -687,7 +687,7 @@ found_device:
  * This searches for sas_device based on sas_address, then return sas_device
  * object.
  */
-static struct _sas_device *
+struct _sas_device *
 mpt3sas_get_sdev_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 {
 	struct _sas_device *sas_device;
@@ -1033,6 +1033,55 @@ mpt3sas_get_pdev_by_idchannel(struct MPT3SAS_ADAPTER *ioc, int id, int channel)
 
 	return pcie_device;
 }
+
+
+struct _pcie_device *
+__mpt3sas_get_pdev_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
+{
+	struct _pcie_device *pcie_device;
+
+	assert_spin_locked(&ioc->pcie_device_lock);
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_list, list)
+		if (pcie_device->handle == handle)
+			goto found_device;
+
+	list_for_each_entry(pcie_device, &ioc->pcie_device_init_list, list)
+		if (pcie_device->handle == handle)
+			goto found_device;
+
+	return NULL;
+
+found_device:
+	pcie_device_get(pcie_device);
+	return pcie_device;
+}
+
+
+/**
+ * mpt3sas_get_pdev_by_handle - pcie device search
+ * @ioc: per adapter object
+ * @handle: Firmware device handle
+ *
+ * Context: This function will acquire ioc->pcie_device_lock and will release
+ * before returning the pcie_device object.
+ *
+ * This searches for pcie_device based on handle, then return pcie_device
+ * object.
+ */
+struct _pcie_device *
+mpt3sas_get_pdev_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
+{
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device = __mpt3sas_get_pdev_by_handle(ioc, handle);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	return pcie_device;
+}
+
 /**
  * _scsih_pcie_device_remove - remove pcie_device from list.
  * @ioc: per adapter object
@@ -1078,6 +1127,85 @@ _scsih_pcie_device_remove(struct MPT3SAS_ADAPTER *ioc,
 	}
 }
 /**
+ * _scsih_pcie_device_add - add pcie_device object
+ * @ioc: per adapter object
+ * @pcie_device: pcie_device object
+ *
+ * This is added to the pcie_device_list link list.
+ */
+static void
+_scsih_pcie_device_add(struct MPT3SAS_ADAPTER *ioc,
+	struct _pcie_device *pcie_device)
+{
+	unsigned long flags;
+
+	dewtprintk(ioc, pr_info(MPT3SAS_FMT
+		"%s: handle (0x%04x), wwid(0x%016llx)\n", ioc->name, __func__,
+		pcie_device->handle, (unsigned long long)pcie_device->wwid));
+	if (pcie_device->enclosure_handle != 0)
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+			"%s: enclosure logical id(0x%016llx), slot( %d)\n",
+			ioc->name, __func__,
+			(unsigned long long)pcie_device->enclosure_logical_id,
+			pcie_device->slot));
+	if (pcie_device->connector_name[0] != '\0')
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+			"%s: enclosure level(0x%04x), connector name( %s)\n",
+			ioc->name, __func__, pcie_device->enclosure_level,
+			pcie_device->connector_name));
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device_get(pcie_device);
+	list_add_tail(&pcie_device->list, &ioc->pcie_device_list);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+	if (scsi_add_device(ioc->shost, PCIE_CHANNEL, pcie_device->id, 0)) {
+		_scsih_pcie_device_remove(ioc, pcie_device);
+	} else if (!pcie_device->starget) {
+		if (!ioc->is_driver_loading) {
+/*TODO-- Need to find out whether this condition will occur or not*/
+			clear_bit(pcie_device->handle, ioc->pend_os_device_add);
+		}
+	} else
+		clear_bit(pcie_device->handle, ioc->pend_os_device_add);
+}
+
+/*
+ * _scsih_pcie_device_init_add - insert pcie_device to the init list.
+ * @ioc: per adapter object
+ * @pcie_device: the pcie_device object
+ * Context: This function will acquire ioc->pcie_device_lock.
+ *
+ * Adding new object at driver load time to the ioc->pcie_device_init_list.
+ */
+static void
+_scsih_pcie_device_init_add(struct MPT3SAS_ADAPTER *ioc,
+				struct _pcie_device *pcie_device)
+{
+	unsigned long flags;
+
+	dewtprintk(ioc, pr_info(MPT3SAS_FMT
+		"%s: handle (0x%04x), wwid(0x%016llx)\n", ioc->name, __func__,
+		pcie_device->handle, (unsigned long long)pcie_device->wwid));
+	if (pcie_device->enclosure_handle != 0)
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+			"%s: enclosure logical id(0x%016llx), slot( %d)\n",
+			ioc->name, __func__,
+			(unsigned long long)pcie_device->enclosure_logical_id,
+			pcie_device->slot));
+	if (pcie_device->connector_name[0] != '\0')
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT
+			"%s: enclosure level(0x%04x), connector name( %s)\n",
+			ioc->name, __func__, pcie_device->enclosure_level,
+			pcie_device->connector_name));
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device_get(pcie_device);
+	list_add_tail(&pcie_device->list, &ioc->pcie_device_init_list);
+	_scsih_determine_boot_device(ioc, pcie_device, PCIE_CHANNEL);
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+}
+/**
  * _scsih_raid_device_find_by_id - raid device search
  * @ioc: per adapter object
  * @id: sas device target id
@@ -1288,6 +1416,23 @@ _scsih_is_end_device(u32 device_info)
 }
 
 /**
+ * _scsih_is_nvme_device - determines if device is an nvme device
+ * @device_info: bitfield providing information about the device.
+ * Context: none
+ *
+ * Returns 1 if nvme device.
+ */
+static int
+_scsih_is_nvme_device(u32 device_info)
+{
+	if ((device_info & MPI26_PCIE_DEVINFO_MASK_DEVICE_TYPE)
+					== MPI26_PCIE_DEVINFO_NVME)
+		return 1;
+	else
+		return 0;
+}
+
+/**
  * _scsih_scsi_lookup_get - returns scmd entry
  * @ioc: per adapter object
  * @smid: system request message index
@@ -6337,6 +6482,314 @@ out:
 }
 
 /**
+ * _scsih_check_pcie_access_status - check access flags
+ * @ioc: per adapter object
+ * @wwid: wwid
+ * @handle: sas device handle
+ * @access_flags: errors returned during discovery of the device
+ *
+ * Return 0 for success, else failure
+ */
+static u8
+_scsih_check_pcie_access_status(struct MPT3SAS_ADAPTER *ioc, u64 wwid,
+	u16 handle, u8 access_status)
+{
+	u8 rc = 1;
+	char *desc = NULL;
+
+	switch (access_status) {
+	case MPI26_PCIEDEV0_ASTATUS_NO_ERRORS:
+	case MPI26_PCIEDEV0_ASTATUS_NEEDS_INITIALIZATION:
+		rc = 0;
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_CAPABILITY_FAILED:
+		desc = "PCIe device capability failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_DEVICE_BLOCKED:
+		desc = "PCIe device blocked";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_MEMORY_SPACE_ACCESS_FAILED:
+		desc = "PCIe device mem space access failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_UNSUPPORTED_DEVICE:
+		desc = "PCIe device unsupported";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_MSIX_REQUIRED:
+		desc = "PCIe device MSIx Required";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_INIT_FAIL_MAX:
+		desc = "PCIe device init fail max";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_UNKNOWN:
+		desc = "PCIe device status unknown";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_READY_TIMEOUT:
+		desc = "nvme ready timeout";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_DEVCFG_UNSUPPORTED:
+		desc = "nvme device configuration unsupported";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_IDENTIFY_FAILED:
+		desc = "nvme identify failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_QCONFIG_FAILED:
+		desc = "nvme qconfig failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_QCREATION_FAILED:
+		desc = "nvme qcreation failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_EVENTCFG_FAILED:
+		desc = "nvme eventcfg failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_GET_FEATURE_STAT_FAILED:
+		desc = "nvme get feature stat failed";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_IDLE_TIMEOUT:
+		desc = "nvme idle timeout";
+		break;
+	case MPI26_PCIEDEV0_ASTATUS_NVME_FAILURE_STATUS:
+		desc = "nvme failure status";
+		break;
+	default:
+		pr_err(MPT3SAS_FMT
+		    " NVMe discovery error(0x%02x): wwid(0x%016llx),"
+			"handle(0x%04x)\n", ioc->name, access_status,
+			(unsigned long long)wwid, handle);
+		return rc;
+	}
+
+	if (!rc)
+		return rc;
+
+	pr_info(MPT3SAS_FMT
+		"NVMe discovery error(%s): wwid(0x%016llx), handle(0x%04x)\n",
+			ioc->name, desc,
+			(unsigned long long)wwid, handle);
+	return rc;
+}
+/**
+ * _scsih_pcie_check_device - checking device responsiveness
+ * @ioc: per adapter object
+ * @handle: attached device handle
+ *
+ * Returns nothing.
+ */
+static void
+_scsih_pcie_check_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
+{
+	Mpi2ConfigReply_t mpi_reply;
+	Mpi26PCIeDevicePage0_t pcie_device_pg0;
+	u32 ioc_status;
+	struct _pcie_device *pcie_device;
+	u64 wwid;
+	unsigned long flags;
+	struct scsi_target *starget;
+	struct MPT3SAS_TARGET *sas_target_priv_data;
+	u32 device_info;
+
+	if ((mpt3sas_config_get_pcie_device_pg0(ioc, &mpi_reply,
+		&pcie_device_pg0, MPI26_PCIE_DEVICE_PGAD_FORM_HANDLE, handle)))
+		return;
+
+	ioc_status = le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK;
+	if (ioc_status != MPI2_IOCSTATUS_SUCCESS)
+		return;
+
+	/* check if this is end device */
+	device_info = le32_to_cpu(pcie_device_pg0.DeviceInfo);
+	if (!(_scsih_is_nvme_device(device_info)))
+		return;
+
+	wwid = le64_to_cpu(pcie_device_pg0.WWID);
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	pcie_device = __mpt3sas_get_pdev_by_wwid(ioc, wwid);
+
+	if (!pcie_device) {
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		return;
+	}
+
+	if (unlikely(pcie_device->handle != handle)) {
+		starget = pcie_device->starget;
+		sas_target_priv_data = starget->hostdata;
+		starget_printk(KERN_INFO, starget,
+		    "handle changed from(0x%04x) to (0x%04x)!!!\n",
+		    pcie_device->handle, handle);
+		sas_target_priv_data->handle = handle;
+		pcie_device->handle = handle;
+
+		if (le32_to_cpu(pcie_device_pg0.Flags) &
+		    MPI26_PCIEDEV0_FLAGS_ENCL_LEVEL_VALID) {
+			pcie_device->enclosure_level =
+			    pcie_device_pg0.EnclosureLevel;
+			memcpy(&pcie_device->connector_name[0],
+			    &pcie_device_pg0.ConnectorName[0], 4);
+		} else {
+			pcie_device->enclosure_level = 0;
+			pcie_device->connector_name[0] = '\0';
+		}
+	}
+
+	/* check if device is present */
+	if (!(le32_to_cpu(pcie_device_pg0.Flags) &
+	    MPI26_PCIEDEV0_FLAGS_DEVICE_PRESENT)) {
+		pr_info(MPT3SAS_FMT
+		    "device is not present handle(0x%04x), flags!!!\n",
+		    ioc->name, handle);
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		pcie_device_put(pcie_device);
+		return;
+	}
+
+	/* check if there were any issues with discovery */
+	if (_scsih_check_pcie_access_status(ioc, wwid, handle,
+	    pcie_device_pg0.AccessStatus)) {
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		pcie_device_put(pcie_device);
+		return;
+	}
+
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+	pcie_device_put(pcie_device);
+
+	_scsih_ublock_io_device(ioc, wwid);
+
+	return;
+}
+
+/**
+ * _scsih_pcie_add_device -  creating pcie device object
+ * @ioc: per adapter object
+ * @handle: pcie device handle
+ *
+ * Creating end device object, stored in ioc->pcie_device_list.
+ *
+ * Return 1 means queue the event later, 0 means complete the event
+ */
+static int
+_scsih_pcie_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
+{
+	Mpi26PCIeDevicePage0_t pcie_device_pg0;
+	Mpi26PCIeDevicePage2_t pcie_device_pg2;
+	Mpi2ConfigReply_t mpi_reply;
+	Mpi2SasEnclosurePage0_t enclosure_pg0;
+	struct _pcie_device *pcie_device;
+	u32 pcie_device_type;
+	u32 ioc_status;
+	u64 wwid;
+
+	if ((mpt3sas_config_get_pcie_device_pg0(ioc, &mpi_reply,
+	    &pcie_device_pg0, MPI26_PCIE_DEVICE_PGAD_FORM_HANDLE, handle))) {
+		pr_err(MPT3SAS_FMT "failure at %s:%d/%s()!\n",
+		    ioc->name, __FILE__, __LINE__, __func__);
+		return 0;
+	}
+	ioc_status = le16_to_cpu(mpi_reply.IOCStatus) &
+	    MPI2_IOCSTATUS_MASK;
+	if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
+		pr_err(MPT3SAS_FMT
+		    "failure at %s:%d/%s()!\n",
+		    ioc->name, __FILE__, __LINE__, __func__);
+		return 0;
+	}
+
+	set_bit(handle, ioc->pend_os_device_add);
+	wwid = le64_to_cpu(pcie_device_pg0.WWID);
+
+	/* check if device is present */
+	if (!(le32_to_cpu(pcie_device_pg0.Flags) &
+		MPI26_PCIEDEV0_FLAGS_DEVICE_PRESENT)) {
+		pr_err(MPT3SAS_FMT
+		    "device is not present handle(0x04%x)!!!\n",
+		    ioc->name, handle);
+		return 0;
+	}
+
+	/* check if there were any issues with discovery */
+	if (_scsih_check_pcie_access_status(ioc, wwid, handle,
+	    pcie_device_pg0.AccessStatus))
+		return 0;
+
+	if (!(_scsih_is_nvme_device(le32_to_cpu(pcie_device_pg0.DeviceInfo))))
+		return 0;
+
+	pcie_device = mpt3sas_get_pdev_by_wwid(ioc, wwid);
+	if (pcie_device) {
+		clear_bit(handle, ioc->pend_os_device_add);
+		pcie_device_put(pcie_device);
+		return 0;
+	}
+
+	pcie_device = kzalloc(sizeof(struct _pcie_device), GFP_KERNEL);
+	if (!pcie_device) {
+		pr_err(MPT3SAS_FMT "failure at %s:%d/%s()!\n",
+			ioc->name, __FILE__, __LINE__, __func__);
+		return 0;
+	}
+
+	kref_init(&pcie_device->refcount);
+	pcie_device->id = ioc->pcie_target_id++;
+	pcie_device->channel = PCIE_CHANNEL;
+	pcie_device->handle = handle;
+	pcie_device->device_info = le32_to_cpu(pcie_device_pg0.DeviceInfo);
+	pcie_device->wwid = wwid;
+	pcie_device->port_num = pcie_device_pg0.PortNum;
+	pcie_device->fast_path = (le32_to_cpu(pcie_device_pg0.Flags) &
+	    MPI26_PCIEDEV0_FLAGS_FAST_PATH_CAPABLE) ? 1 : 0;
+	pcie_device_type = pcie_device->device_info &
+	    MPI26_PCIE_DEVINFO_MASK_DEVICE_TYPE;
+
+	pcie_device->enclosure_handle =
+	    le16_to_cpu(pcie_device_pg0.EnclosureHandle);
+	if (pcie_device->enclosure_handle != 0)
+		pcie_device->slot = le16_to_cpu(pcie_device_pg0.Slot);
+
+	if (le16_to_cpu(pcie_device_pg0.Flags) &
+	    MPI26_PCIEDEV0_FLAGS_ENCL_LEVEL_VALID) {
+		pcie_device->enclosure_level = pcie_device_pg0.EnclosureLevel;
+		memcpy(&pcie_device->connector_name[0],
+		    &pcie_device_pg0.ConnectorName[0], 4);
+	} else {
+		pcie_device->enclosure_level = 0;
+		pcie_device->connector_name[0] = '\0';
+	}
+
+	/* get enclosure_logical_id */
+	if (pcie_device->enclosure_handle &&
+		!(mpt3sas_config_get_enclosure_pg0(ioc, &mpi_reply,
+			&enclosure_pg0, MPI2_SAS_ENCLOS_PGAD_FORM_HANDLE,
+			pcie_device->enclosure_handle)))
+		pcie_device->enclosure_logical_id =
+			le64_to_cpu(enclosure_pg0.EnclosureLogicalID);
+
+	/*TODO -- Add device name once FW supports it*/
+	if (mpt3sas_config_get_pcie_device_pg2(ioc, &mpi_reply,
+		&pcie_device_pg2, MPI2_SAS_DEVICE_PGAD_FORM_HANDLE, handle)) {
+		pr_err(MPT3SAS_FMT "failure at %s:%d/%s()!\n",
+				ioc->name, __FILE__, __LINE__, __func__);
+		kfree(pcie_device);
+		return 0;
+	}
+
+	ioc_status = le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK;
+	if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
+		pr_err(MPT3SAS_FMT "failure at %s:%d/%s()!\n",
+			ioc->name, __FILE__, __LINE__, __func__);
+		kfree(pcie_device);
+		return 0;
+	}
+	pcie_device->nvme_mdts =
+		le32_to_cpu(pcie_device_pg2.MaximumDataTransferSize);
+
+	if (ioc->wait_for_discovery_to_complete)
+		_scsih_pcie_device_init_add(ioc, pcie_device);
+	else
+		_scsih_pcie_device_add(ioc, pcie_device);
+
+	pcie_device_put(pcie_device);
+	return 0;
+}
+/**
  * _scsih_sas_enclosure_dev_status_change_event_debug - debug for enclosure
  * event
  * @ioc: per adapter object
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:33 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:33 +0530
Subject: [PATCH 07/13] mpt3sas: Set NVMe device queue depth as 128
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-8-git-send-email-suganath-prabu.subramani@broadcom.com>

Sets nvme device queue depth, name and displays device capabilities

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  2 +-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 50 ++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index a4ec459..2d005af 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -117,7 +117,7 @@
 
 #define MPT3SAS_RAID_MAX_SECTORS	8192
 #define MPT3SAS_HOST_PAGE_SIZE_4K	12
-
+#define MPT3SAS_NVME_QUEUE_DEPTH	128
 #define MPT_NAME_LENGTH			32	/* generic length of strings */
 #define MPT_STRING_LENGTH		64
 
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 21260bd..9006a60 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2324,6 +2324,7 @@ scsih_slave_configure(struct scsi_device *sdev)
 	struct MPT3SAS_DEVICE *sas_device_priv_data;
 	struct MPT3SAS_TARGET *sas_target_priv_data;
 	struct _sas_device *sas_device;
+	struct _pcie_device *pcie_device;
 	struct _raid_device *raid_device;
 	unsigned long flags;
 	int qdepth;
@@ -2454,6 +2455,55 @@ scsih_slave_configure(struct scsi_device *sdev)
 		}
 	}
 
+	/* PCIe handling */
+	if (sas_target_priv_data->flags & MPT_TARGET_FLAGS_PCIE_DEVICE) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_by_wwid(ioc,
+				sas_device_priv_data->sas_target->sas_address);
+		if (!pcie_device) {
+			spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+			dfailprintk(ioc, pr_warn(MPT3SAS_FMT
+				"failure at %s:%d/%s()!\n", ioc->name, __FILE__,
+				__LINE__, __func__));
+			return 1;
+		}
+
+		qdepth = MPT3SAS_NVME_QUEUE_DEPTH;
+		ds = "NVMe";
+		sdev_printk(KERN_INFO, sdev,
+			"%s: handle(0x%04x), wwid(0x%016llx), port(%d)\n",
+			ds, handle, (unsigned long long)pcie_device->wwid,
+			pcie_device->port_num);
+		if (pcie_device->enclosure_handle != 0)
+			sdev_printk(KERN_INFO, sdev,
+			"%s: enclosure logical id(0x%016llx), slot(%d)\n",
+			ds,
+			(unsigned long long)pcie_device->enclosure_logical_id,
+			pcie_device->slot);
+		if (pcie_device->connector_name[0] != '\0')
+			sdev_printk(KERN_INFO, sdev,
+				"%s: enclosure level(0x%04x),"
+				"connector name( %s)\n", ds,
+				pcie_device->enclosure_level,
+				pcie_device->connector_name);
+		pcie_device_put(pcie_device);
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+		scsih_change_queue_depth(sdev, qdepth);
+
+		if (pcie_device->nvme_mdts)
+			blk_queue_max_hw_sectors(sdev->request_queue,
+					pcie_device->nvme_mdts/512);
+		/* Enable QUEUE_FLAG_NOMERGES flag, so that IOs won't be
+		 ** merged and can eliminate holes created during merging
+		 ** operation.
+		 **/
+		queue_flag_set_unlocked(QUEUE_FLAG_NOMERGES,
+				sdev->request_queue);
+		blk_queue_virt_boundary(sdev->request_queue,
+				ioc->page_size - 1);
+		return 0;
+	}
+
 	spin_lock_irqsave(&ioc->sas_device_lock, flags);
 	sas_device = __mpt3sas_get_sdev_by_addr(ioc,
 	   sas_device_priv_data->sas_target->sas_address);
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:32 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:32 +0530
Subject: [PATCH 06/13] mpt3sas: Handle NVMe PCIe device related events
 generated from firmware.
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-7-git-send-email-suganath-prabu.subramani@broadcom.com>

* The controller firmware sends separate events for NVMe devices and
PCIe switches similar to existing SAS events.

* NVMe device detection, addition and removal are reported by the
firmware through PCIe Topology Change list events.

* The PCIe device state change events are sent when the firmware
detects any abnormal conditions with a NVMe device or switch.

* The enumeration event are sent when the firmware starts PCIe device
enumeration and stops.

* This patch has the code change to handle the events and add/remove
NVMe devices in driver's inventory.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c  |  30 ++-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 471 ++++++++++++++++++++++++++++++++++-
 2 files changed, 495 insertions(+), 6 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index b0a75c6..0da639d 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -663,6 +663,26 @@ _base_display_event_data(struct MPT3SAS_ADAPTER *ioc,
 	case MPI2_EVENT_ACTIVE_CABLE_EXCEPTION:
 		desc = "Cable Event";
 		break;
+	case MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE:
+		desc = "PCIE Device Status Change";
+		break;
+	case MPI2_EVENT_PCIE_ENUMERATION:
+	{
+		Mpi26EventDataPCIeEnumeration_t *event_data =
+			(Mpi26EventDataPCIeEnumeration_t *)mpi_reply->EventData;
+		pr_info(MPT3SAS_FMT "PCIE Enumeration: (%s)", ioc->name,
+			   (event_data->ReasonCode ==
+				MPI26_EVENT_PCIE_ENUM_RC_STARTED) ?
+				"start" : "stop");
+		if (event_data->EnumerationStatus)
+			pr_info("enumeration_status(0x%08x)",
+				   le32_to_cpu(event_data->EnumerationStatus));
+		pr_info("\n");
+		return;
+	}
+	case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
+		desc = "PCIE Topology Change List";
+		break;
 	}
 
 	if (!desc)
@@ -6125,7 +6145,15 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc)
 	_base_unmask_events(ioc, MPI2_EVENT_LOG_ENTRY_ADDED);
 	_base_unmask_events(ioc, MPI2_EVENT_TEMP_THRESHOLD);
 	_base_unmask_events(ioc, MPI2_EVENT_ACTIVE_CABLE_EXCEPTION);
-
+	if (ioc->hba_mpi_version_belonged == MPI26_VERSION) {
+		if (ioc->is_gen35_ioc) {
+			_base_unmask_events(ioc,
+				MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE);
+			_base_unmask_events(ioc, MPI2_EVENT_PCIE_ENUMERATION);
+			_base_unmask_events(ioc,
+				MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST);
+		}
+	}
 	r = _base_make_ioc_operational(ioc);
 	if (r)
 		goto out_free_resources;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index d83560c..21260bd 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -75,6 +75,8 @@ static int _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 static int _scsih_pcie_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle);
 static void _scsih_pcie_device_remove_from_sml(struct MPT3SAS_ADAPTER *ioc,
 	struct _pcie_device *pcie_device);
+static void
+_scsih_pcie_check_device(struct MPT3SAS_ADAPTER *ioc, u16 handle);
 static u8 _scsih_check_for_pending_tm(struct MPT3SAS_ADAPTER *ioc, u16 smid);
 
 /* global parameters */
@@ -3476,8 +3478,6 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 	struct _sas_device *sas_device;
 
 	sas_device = mpt3sas_get_sdev_by_handle(ioc, handle);
-	if (!sas_device)
-		return;
 
 	shost_for_each_device(sdev, ioc->shost) {
 		sas_device_priv_data = sdev->hostdata;
@@ -3487,7 +3487,7 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 			continue;
 		if (sas_device_priv_data->block)
 			continue;
-		if (sas_device->pend_sas_rphy_add)
+		if (sas_device && sas_device->pend_sas_rphy_add)
 			continue;
 		if (sas_device_priv_data->ignore_delay_remove) {
 			sdev_printk(KERN_INFO, sdev,
@@ -3498,7 +3498,8 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 		_scsih_internal_device_block(sdev, sas_device_priv_data);
 	}
 
-	sas_device_put(sas_device);
+	if (sas_device)
+		sas_device_put(sas_device);
 }
 
 /**
@@ -3582,6 +3583,33 @@ _scsih_block_io_to_children_attached_directly(struct MPT3SAS_ADAPTER *ioc,
 }
 
 /**
+ * _scsih_block_io_to_pcie_children_attached_directly
+ * @ioc: per adapter object
+ * @event_data: topology change event data
+ *
+ * This routine set sdev state to SDEV_BLOCK for all devices
+ * direct attached during device pull/reconnect.
+ */
+static void
+_scsih_block_io_to_pcie_children_attached_directly(struct MPT3SAS_ADAPTER *ioc,
+		Mpi26EventDataPCIeTopologyChangeList_t *event_data)
+{
+	int i;
+	u16 handle;
+	u16 reason_code;
+
+	for (i = 0; i < event_data->NumEntries; i++) {
+		handle =
+			le16_to_cpu(event_data->PortEntry[i].AttachedDevHandle);
+		if (!handle)
+			continue;
+		reason_code = event_data->PortEntry[i].PortStatus;
+		if (reason_code ==
+				MPI26_EVENT_PCIE_TOPO_PS_DELAY_NOT_RESPONDING)
+			_scsih_block_io_device(ioc, handle);
+	}
+}
+/**
  * _scsih_tm_tr_send - send task management request
  * @ioc: per adapter object
  * @handle: device handle
@@ -4190,6 +4218,81 @@ _scsih_check_topo_delete_events(struct MPT3SAS_ADAPTER *ioc,
 }
 
 /**
+ * _scsih_check_pcie_topo_remove_events - sanity check on topo
+ * events
+ * @ioc: per adapter object
+ * @event_data: the event data payload
+ *
+ * This handles the case where driver receives multiple switch
+ * or device add and delete events in a single shot.  When there
+ * is a delete event the routine will void any pending add
+ * events waiting in the event queue.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_check_pcie_topo_remove_events(struct MPT3SAS_ADAPTER *ioc,
+	Mpi26EventDataPCIeTopologyChangeList_t *event_data)
+{
+	struct fw_event_work *fw_event;
+	Mpi26EventDataPCIeTopologyChangeList_t *local_event_data;
+	unsigned long flags;
+	int i, reason_code;
+	u16 handle, switch_handle;
+
+	for (i = 0; i < event_data->NumEntries; i++) {
+		handle =
+			le16_to_cpu(event_data->PortEntry[i].AttachedDevHandle);
+		if (!handle)
+			continue;
+		reason_code = event_data->PortEntry[i].PortStatus;
+		if (reason_code == MPI26_EVENT_PCIE_TOPO_PS_NOT_RESPONDING)
+			_scsih_tm_tr_send(ioc, handle);
+	}
+
+	switch_handle = le16_to_cpu(event_data->SwitchDevHandle);
+	if (!switch_handle) {
+		_scsih_block_io_to_pcie_children_attached_directly(
+							ioc, event_data);
+		return;
+	}
+    /* TODO We are not supporting cascaded PCIe Switch removal yet*/
+	if ((event_data->SwitchStatus
+		== MPI26_EVENT_PCIE_TOPO_SS_DELAY_NOT_RESPONDING) ||
+		(event_data->SwitchStatus ==
+					MPI26_EVENT_PCIE_TOPO_SS_RESPONDING))
+		_scsih_block_io_to_pcie_children_attached_directly(
+							ioc, event_data);
+
+	if (event_data->SwitchStatus != MPI2_EVENT_SAS_TOPO_ES_NOT_RESPONDING)
+		return;
+
+	/* mark ignore flag for pending events */
+	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	list_for_each_entry(fw_event, &ioc->fw_event_list, list) {
+		if (fw_event->event != MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST ||
+			fw_event->ignore)
+			continue;
+		local_event_data =
+			(Mpi26EventDataPCIeTopologyChangeList_t *)
+			fw_event->event_data;
+		if (local_event_data->SwitchStatus ==
+		    MPI2_EVENT_SAS_TOPO_ES_ADDED ||
+		    local_event_data->SwitchStatus ==
+		    MPI2_EVENT_SAS_TOPO_ES_RESPONDING) {
+			if (le16_to_cpu(local_event_data->SwitchDevHandle) ==
+				switch_handle) {
+				dewtprintk(ioc, pr_info(MPT3SAS_FMT
+					"setting ignoring flag for switch event\n",
+					ioc->name));
+				fw_event->ignore = 1;
+			}
+		}
+	}
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+}
+
+/**
  * _scsih_set_volume_delete_flag - setting volume delete flag
  * @ioc: per adapter object
  * @handle: device handle
@@ -6515,9 +6618,9 @@ out:
 		sas_device_put(sas_device);
 
 	spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
-
 }
 
+
 /**
  * _scsih_check_pcie_access_status - check access flags
  * @ioc: per adapter object
@@ -6903,6 +7006,319 @@ _scsih_pcie_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 	pcie_device_put(pcie_device);
 	return 0;
 }
+
+/**
+ * _scsih_pcie_topology_change_event_debug - debug for topology
+ * event
+ * @ioc: per adapter object
+ * @event_data: event data payload
+ * Context: user.
+ */
+static void
+_scsih_pcie_topology_change_event_debug(struct MPT3SAS_ADAPTER *ioc,
+	Mpi26EventDataPCIeTopologyChangeList_t *event_data)
+{
+	int i;
+	u16 handle;
+	u16 reason_code;
+	u8 port_number;
+	char *status_str = NULL;
+	u8 link_rate, prev_link_rate;
+
+	switch (event_data->SwitchStatus) {
+	case MPI26_EVENT_PCIE_TOPO_SS_ADDED:
+		status_str = "add";
+		break;
+	case MPI26_EVENT_PCIE_TOPO_SS_NOT_RESPONDING:
+		status_str = "remove";
+		break;
+	case MPI26_EVENT_PCIE_TOPO_SS_RESPONDING:
+	case 0:
+		status_str =  "responding";
+		break;
+	case MPI26_EVENT_PCIE_TOPO_SS_DELAY_NOT_RESPONDING:
+		status_str = "remove delay";
+		break;
+	default:
+		status_str = "unknown status";
+		break;
+	}
+	pr_info(MPT3SAS_FMT "pcie topology change: (%s)\n",
+		ioc->name, status_str);
+	pr_info("\tswitch_handle(0x%04x), enclosure_handle(0x%04x)"
+		"start_port(%02d), count(%d)\n",
+		le16_to_cpu(event_data->SwitchDevHandle),
+		le16_to_cpu(event_data->EnclosureHandle),
+		event_data->StartPortNum, event_data->NumEntries);
+	for (i = 0; i < event_data->NumEntries; i++) {
+		handle =
+			le16_to_cpu(event_data->PortEntry[i].AttachedDevHandle);
+		if (!handle)
+			continue;
+		port_number = event_data->StartPortNum + i;
+		reason_code = event_data->PortEntry[i].PortStatus;
+		switch (reason_code) {
+		case MPI26_EVENT_PCIE_TOPO_PS_DEV_ADDED:
+			status_str = "target add";
+			break;
+		case MPI26_EVENT_PCIE_TOPO_PS_NOT_RESPONDING:
+			status_str = "target remove";
+			break;
+		case MPI26_EVENT_PCIE_TOPO_PS_DELAY_NOT_RESPONDING:
+			status_str = "delay target remove";
+			break;
+		case MPI26_EVENT_PCIE_TOPO_PS_PORT_CHANGED:
+			status_str = "link rate change";
+			break;
+		case MPI26_EVENT_PCIE_TOPO_PS_NO_CHANGE:
+			status_str = "target responding";
+			break;
+		default:
+			status_str = "unknown";
+			break;
+		}
+		link_rate = event_data->PortEntry[i].CurrentPortInfo &
+			MPI26_EVENT_PCIE_TOPO_PI_RATE_MASK;
+		prev_link_rate = event_data->PortEntry[i].PreviousPortInfo &
+			MPI26_EVENT_PCIE_TOPO_PI_RATE_MASK;
+		pr_info("\tport(%02d), attached_handle(0x%04x): %s:"
+			" link rate: new(0x%02x), old(0x%02x)\n", port_number,
+			handle, status_str, link_rate, prev_link_rate);
+	}
+}
+
+/**
+ * _scsih_pcie_topology_change_event - handle PCIe topology
+ *  changes
+ * @ioc: per adapter object
+ * @fw_event: The fw_event_work object
+ * Context: user.
+ *
+ */
+static int
+_scsih_pcie_topology_change_event(struct MPT3SAS_ADAPTER *ioc,
+	struct fw_event_work *fw_event)
+{
+	int i;
+	u16 handle;
+	u16 reason_code;
+	u8 link_rate, prev_link_rate;
+	unsigned long flags;
+	int rc;
+	int requeue_event;
+	Mpi26EventDataPCIeTopologyChangeList_t *event_data =
+		(Mpi26EventDataPCIeTopologyChangeList_t *) fw_event->event_data;
+	struct _pcie_device *pcie_device;
+
+	if (ioc->logging_level & MPT_DEBUG_EVENT_WORK_TASK)
+		_scsih_pcie_topology_change_event_debug(ioc, event_data);
+
+	if (ioc->shost_recovery || ioc->remove_host ||
+		ioc->pci_error_recovery)
+		return 0;
+
+	if (fw_event->ignore) {
+		dewtprintk(ioc, pr_info(MPT3SAS_FMT "ignoring switch event\n",
+			ioc->name));
+		return 0;
+	}
+
+	/* handle siblings events */
+	for (i = 0; i < event_data->NumEntries; i++) {
+		if (fw_event->ignore) {
+			dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				"ignoring switch event\n", ioc->name));
+			return 0;
+		}
+		if (ioc->remove_host || ioc->pci_error_recovery)
+			return 0;
+		reason_code = event_data->PortEntry[i].PortStatus;
+		handle =
+			le16_to_cpu(event_data->PortEntry[i].AttachedDevHandle);
+		if (!handle)
+			continue;
+
+		link_rate = event_data->PortEntry[i].CurrentPortInfo
+			& MPI26_EVENT_PCIE_TOPO_PI_RATE_MASK;
+		prev_link_rate = event_data->PortEntry[i].PreviousPortInfo
+			& MPI26_EVENT_PCIE_TOPO_PI_RATE_MASK;
+
+		switch (reason_code) {
+		case MPI26_EVENT_PCIE_TOPO_PS_PORT_CHANGED:
+			if (ioc->shost_recovery)
+				break;
+			if (link_rate == prev_link_rate)
+				break;
+			if (link_rate < MPI26_EVENT_PCIE_TOPO_PI_RATE_2_5)
+				break;
+
+			_scsih_pcie_check_device(ioc, handle);
+
+			/* This code after this point handles the test case
+			 * where a device has been added, however its returning
+			 * BUSY for sometime.  Then before the Device Missing
+			 * Delay expires and the device becomes READY, the
+			 * device is removed and added back.
+			 */
+			spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+			pcie_device = __mpt3sas_get_pdev_by_handle(ioc, handle);
+			spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
+			if (pcie_device) {
+				pcie_device_put(pcie_device);
+				break;
+			}
+
+			if (!test_bit(handle, ioc->pend_os_device_add))
+				break;
+
+			dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				"handle(0x%04x) device not found: convert "
+				"event to a device add\n", ioc->name, handle));
+			event_data->PortEntry[i].PortStatus &= 0xF0;
+			event_data->PortEntry[i].PortStatus |=
+				MPI26_EVENT_PCIE_TOPO_PS_DEV_ADDED;
+		case MPI26_EVENT_PCIE_TOPO_PS_DEV_ADDED:
+			if (ioc->shost_recovery)
+				break;
+			if (link_rate < MPI26_EVENT_PCIE_TOPO_PI_RATE_2_5)
+				break;
+
+			rc = _scsih_pcie_add_device(ioc, handle);
+			if (!rc) {
+				/* mark entry vacant */
+				/* TODO This needs to be reviewed and fixed,
+				 * we dont have an entry
+				 * to make an event void like vacant
+				 */
+				event_data->PortEntry[i].PortStatus |=
+					MPI26_EVENT_PCIE_TOPO_PS_NO_CHANGE;
+			}
+			break;
+		case MPI26_EVENT_PCIE_TOPO_PS_NOT_RESPONDING:
+			_scsih_pcie_device_remove_by_handle(ioc, handle);
+			break;
+		}
+	}
+	return requeue_event;
+}
+
+/**
+ * _scsih_pcie_device_status_change_event_debug - debug for
+ * device event
+ * @event_data: event data payload
+ * Context: user.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_pcie_device_status_change_event_debug(struct MPT3SAS_ADAPTER *ioc,
+	Mpi26EventDataPCIeDeviceStatusChange_t *event_data)
+{
+	char *reason_str = NULL;
+
+	switch (event_data->ReasonCode) {
+	case MPI26_EVENT_PCIDEV_STAT_RC_SMART_DATA:
+		reason_str = "smart data";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_UNSUPPORTED:
+		reason_str = "unsupported device discovered";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_INTERNAL_DEVICE_RESET:
+		reason_str = "internal device reset";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_TASK_ABORT_INTERNAL:
+		reason_str = "internal task abort";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_ABORT_TASK_SET_INTERNAL:
+		reason_str = "internal task abort set";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_CLEAR_TASK_SET_INTERNAL:
+		reason_str = "internal clear task set";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_QUERY_TASK_INTERNAL:
+		reason_str = "internal query task";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_DEV_INIT_FAILURE:
+		reason_str = "device init failure";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_CMP_INTERNAL_DEV_RESET:
+		reason_str = "internal device reset complete";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_CMP_TASK_ABORT_INTERNAL:
+		reason_str = "internal task abort complete";
+		break;
+	case MPI26_EVENT_PCIDEV_STAT_RC_ASYNC_NOTIFICATION:
+		reason_str = "internal async notification";
+		break;
+	default:
+		reason_str = "unknown reason";
+		break;
+	}
+
+	pr_info(MPT3SAS_FMT "PCIE device status change: (%s)\n"
+		"\thandle(0x%04x), WWID(0x%016llx), tag(%d)",
+		ioc->name, reason_str, le16_to_cpu(event_data->DevHandle),
+		(unsigned long long)le64_to_cpu(event_data->WWID),
+		le16_to_cpu(event_data->TaskTag));
+	if (event_data->ReasonCode == MPI26_EVENT_PCIDEV_STAT_RC_SMART_DATA)
+		pr_info(MPT3SAS_FMT ", ASC(0x%x), ASCQ(0x%x)\n", ioc->name,
+			event_data->ASC, event_data->ASCQ);
+	pr_info("\n");
+}
+
+/**
+ * _scsih_pcie_device_status_change_event - handle device status
+ * change
+ * @ioc: per adapter object
+ * @fw_event: The fw_event_work object
+ * Context: user.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_pcie_device_status_change_event(struct MPT3SAS_ADAPTER *ioc,
+	struct fw_event_work *fw_event)
+{
+	struct MPT3SAS_TARGET *target_priv_data;
+	struct _pcie_device *pcie_device;
+	u64 wwid;
+	unsigned long flags;
+	Mpi26EventDataPCIeDeviceStatusChange_t *event_data =
+		(Mpi26EventDataPCIeDeviceStatusChange_t *)fw_event->event_data;
+	if (ioc->logging_level & MPT_DEBUG_EVENT_WORK_TASK)
+		_scsih_pcie_device_status_change_event_debug(ioc,
+			event_data);
+
+	if (event_data->ReasonCode !=
+		MPI26_EVENT_PCIDEV_STAT_RC_INTERNAL_DEVICE_RESET &&
+		event_data->ReasonCode !=
+		MPI26_EVENT_PCIDEV_STAT_RC_CMP_INTERNAL_DEV_RESET)
+		return;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	wwid = le64_to_cpu(event_data->WWID);
+	pcie_device = __mpt3sas_get_pdev_by_wwid(ioc, wwid);
+
+	if (!pcie_device || !pcie_device->starget)
+		goto out;
+
+	target_priv_data = pcie_device->starget->hostdata;
+	if (!target_priv_data)
+		goto out;
+
+	if (event_data->ReasonCode ==
+		MPI26_EVENT_PCIDEV_STAT_RC_INTERNAL_DEVICE_RESET)
+		target_priv_data->tm_busy = 1;
+	else
+		target_priv_data->tm_busy = 0;
+out:
+	if (pcie_device)
+		pcie_device_put(pcie_device);
+
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+}
+
 /**
  * _scsih_sas_enclosure_dev_status_change_event_debug - debug for enclosure
  * event
@@ -7154,6 +7570,34 @@ _scsih_sas_discovery_event(struct MPT3SAS_ADAPTER *ioc,
 }
 
 /**
+ * _scsih_pcie_enumeration_event - handle enumeration events
+ * @ioc: per adapter object
+ * @fw_event: The fw_event_work object
+ * Context: user.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_pcie_enumeration_event(struct MPT3SAS_ADAPTER *ioc,
+	struct fw_event_work *fw_event)
+{
+	Mpi26EventDataPCIeEnumeration_t *event_data =
+		(Mpi26EventDataPCIeEnumeration_t *)fw_event->event_data;
+
+	if (ioc->logging_level & MPT_DEBUG_EVENT_WORK_TASK) {
+		pr_info(MPT3SAS_FMT "pcie enumeration event: (%s) Flag 0x%02x",
+			ioc->name,
+			((event_data->ReasonCode ==
+			MPI26_EVENT_PCIE_ENUM_RC_STARTED) ?
+			"started" : "completed"), event_data->Flags);
+	if (event_data->EnumerationStatus)
+		pr_info("enumeration_status(0x%08x)",
+		    le32_to_cpu(event_data->EnumerationStatus));
+	pr_info("\n");
+	}
+}
+
+/**
  * _scsih_ir_fastpath - turn on fastpath for IR physdisk
  * @ioc: per adapter object
  * @handle: device handle for physical disk
@@ -8809,6 +9253,16 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 	case MPI2_EVENT_IR_OPERATION_STATUS:
 		_scsih_sas_ir_operation_status_event(ioc, fw_event);
 		break;
+	case MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE:
+		_scsih_pcie_device_status_change_event(ioc, fw_event);
+		break;
+	case MPI2_EVENT_PCIE_ENUMERATION:
+		_scsih_pcie_enumeration_event(ioc, fw_event);
+		break;
+	case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
+		_scsih_pcie_topology_change_event(ioc, fw_event);
+			return;
+	break;
 	}
 out:
 	fw_event_work_put(fw_event);
@@ -8899,6 +9353,11 @@ mpt3sas_scsih_event_callback(struct MPT3SAS_ADAPTER *ioc, u8 msix_index,
 		    (Mpi2EventDataSasTopologyChangeList_t *)
 		    mpi_reply->EventData);
 		break;
+	case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
+	_scsih_check_pcie_topo_remove_events(ioc,
+		    (Mpi26EventDataPCIeTopologyChangeList_t *)
+		    mpi_reply->EventData);
+		break;
 	case MPI2_EVENT_IR_CONFIGURATION_CHANGE_LIST:
 		_scsih_check_ir_config_unhide_events(ioc,
 		    (Mpi2EventDataIrConfigChangeList_t *)
@@ -8961,6 +9420,8 @@ mpt3sas_scsih_event_callback(struct MPT3SAS_ADAPTER *ioc, u8 msix_index,
 	case MPI2_EVENT_SAS_DISCOVERY:
 	case MPI2_EVENT_SAS_ENCL_DEVICE_STATUS_CHANGE:
 	case MPI2_EVENT_IR_PHYSICAL_DISK:
+	case MPI2_EVENT_PCIE_ENUMERATION:
+	case MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE:
 		break;
 
 	case MPI2_EVENT_TEMP_THRESHOLD:
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:34 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:34 +0530
Subject: [PATCH 08/13] mpt3sas: scan and add nvme device after controller reset
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-9-git-send-email-suganath-prabu.subramani@broadcom.com>

After Controller reset, Scan and add nvme device back to the topology.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 194 ++++++++++++++++++++++++++++++++++-
 1 file changed, 190 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 9006a60..81f4d2d 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4887,6 +4887,7 @@ _scsih_scsi_ioc_info(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
 	char *desc_scsi_state = ioc->tmp_string;
 	u32 log_info = le32_to_cpu(mpi_reply->IOCLogInfo);
 	struct _sas_device *sas_device = NULL;
+	struct _pcie_device *pcie_device = NULL;
 	struct scsi_target *starget = scmd->device->sdev_target;
 	struct MPT3SAS_TARGET *priv_target = starget->hostdata;
 	char *device_str = NULL;
@@ -5019,6 +5020,28 @@ _scsih_scsi_ioc_info(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
 	if (priv_target->flags & MPT_TARGET_FLAGS_VOLUME) {
 		pr_warn(MPT3SAS_FMT "\t%s wwid(0x%016llx)\n", ioc->name,
 		    device_str, (unsigned long long)priv_target->sas_address);
+	} else if (priv_target->flags & MPT_TARGET_FLAGS_PCIE_DEVICE) {
+		pcie_device = mpt3sas_get_pdev_from_target(ioc, priv_target);
+		if (pcie_device) {
+			pr_info(MPT3SAS_FMT "\twwid(0x%016llx), port(%d)\n",
+			    ioc->name,
+			    (unsigned long long)pcie_device->wwid,
+			    pcie_device->port_num);
+			if (pcie_device->enclosure_handle != 0)
+				pr_info(MPT3SAS_FMT
+				    "\tenclosure logical id(0x%016llx), "
+				    "slot(%d)\n", ioc->name,
+				    (unsigned long long)
+				    pcie_device->enclosure_logical_id,
+				    pcie_device->slot);
+			if (pcie_device->connector_name[0])
+				pr_info(MPT3SAS_FMT
+				    "\tenclosure level(0x%04x),"
+				    "connector name( %s)\n",
+				    ioc->name, pcie_device->enclosure_level,
+				    pcie_device->connector_name);
+			pcie_device_put(pcie_device);
+		}
 	} else {
 		sas_device = mpt3sas_get_sdev_from_target(ioc, priv_target);
 		if (sas_device) {
@@ -5055,11 +5078,10 @@ _scsih_scsi_ioc_info(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd,
 		struct sense_info data;
 		_scsih_normalize_sense(scmd->sense_buffer, &data);
 		pr_warn(MPT3SAS_FMT
-			"\t[sense_key,asc,ascq]: [0x%02x,0x%02x,0x%02x], count(%d)\n",
-			ioc->name, data.skey,
-		    data.asc, data.ascq, le32_to_cpu(mpi_reply->SenseCount));
+		  "\t[sense_key,asc,ascq]: [0x%02x,0x%02x,0x%02x], count(%d)\n",
+		  ioc->name, data.skey,
+		  data.asc, data.ascq, le32_to_cpu(mpi_reply->SenseCount));
 	}
-
 	if (scsi_state & MPI2_SCSI_STATE_RESPONSE_INFO_VALID) {
 		response_info = le32_to_cpu(mpi_reply->ResponseInfo);
 		response_bytes = (u8 *)&response_info;
@@ -8559,6 +8581,130 @@ _scsih_search_responding_sas_devices(struct MPT3SAS_ADAPTER *ioc)
 }
 
 /**
+ * _scsih_mark_responding_pcie_device - mark a pcie_device as responding
+ * @ioc: per adapter object
+ * @pcie_device_pg0: PCIe Device page 0
+ *
+ * After host reset, find out whether devices are still responding.
+ * Used in _scsih_remove_unresponding_devices.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_mark_responding_pcie_device(struct MPT3SAS_ADAPTER *ioc,
+	Mpi26PCIeDevicePage0_t *pcie_device_pg0)
+{
+	struct MPT3SAS_TARGET *sas_target_priv_data = NULL;
+	struct scsi_target *starget;
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	list_for_each_entry(pcie_device, &ioc->pcie_device_list, list) {
+		if ((pcie_device->wwid == pcie_device_pg0->WWID) &&
+		    (pcie_device->slot == pcie_device_pg0->Slot)) {
+			pcie_device->responding = 1;
+			starget = pcie_device->starget;
+			if (starget && starget->hostdata) {
+				sas_target_priv_data = starget->hostdata;
+				sas_target_priv_data->tm_busy = 0;
+				sas_target_priv_data->deleted = 0;
+			} else
+				sas_target_priv_data = NULL;
+			if (starget) {
+				starget_printk(KERN_INFO, starget,
+				    "handle(0x%04x), wwid(0x%016llx) ",
+				    pcie_device->handle,
+				    (unsigned long long)pcie_device->wwid);
+				if (pcie_device->enclosure_handle != 0)
+					starget_printk(KERN_INFO, starget,
+					    "enclosure logical id(0x%016llx), "
+					    "slot(%d)\n",
+					    (unsigned long long)
+					    pcie_device->enclosure_logical_id,
+					    pcie_device->slot);
+			}
+
+			if (((le32_to_cpu(pcie_device_pg0->Flags)) &
+			    MPI26_PCIEDEV0_FLAGS_ENCL_LEVEL_VALID) &&
+			    (ioc->hba_mpi_version_belonged != MPI2_VERSION)) {
+				pcie_device->enclosure_level =
+				    pcie_device_pg0->EnclosureLevel;
+				memcpy(&pcie_device->connector_name[0],
+				    &pcie_device_pg0->ConnectorName[0], 4);
+			} else {
+				pcie_device->enclosure_level = 0;
+				pcie_device->connector_name[0] = '\0';
+			}
+
+			if (pcie_device->handle == pcie_device_pg0->DevHandle)
+				goto out;
+			pr_info(KERN_INFO "\thandle changed from(0x%04x)!!!\n",
+			    pcie_device->handle);
+			pcie_device->handle = pcie_device_pg0->DevHandle;
+			if (sas_target_priv_data)
+				sas_target_priv_data->handle =
+				    pcie_device_pg0->DevHandle;
+			goto out;
+		}
+	}
+
+ out:
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+}
+
+/**
+ * _scsih_search_responding_pcie_devices -
+ * @ioc: per adapter object
+ *
+ * After host reset, find out whether devices are still responding.
+ * If not remove.
+ *
+ * Return nothing.
+ */
+static void
+_scsih_search_responding_pcie_devices(struct MPT3SAS_ADAPTER *ioc)
+{
+	Mpi26PCIeDevicePage0_t pcie_device_pg0;
+	Mpi2ConfigReply_t mpi_reply;
+	u16 ioc_status;
+	u16 handle;
+	u32 device_info;
+
+	pr_info(MPT3SAS_FMT "search for end-devices: start\n", ioc->name);
+
+	if (list_empty(&ioc->pcie_device_list))
+		goto out;
+
+	handle = 0xFFFF;
+	while (!(mpt3sas_config_get_pcie_device_pg0(ioc, &mpi_reply,
+		&pcie_device_pg0, MPI26_PCIE_DEVICE_PGAD_FORM_GET_NEXT_HANDLE,
+		handle))) {
+		ioc_status = le16_to_cpu(mpi_reply.IOCStatus) &
+		    MPI2_IOCSTATUS_MASK;
+		if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
+			pr_info(MPT3SAS_FMT "\tbreak from %s: "
+			    "ioc_status(0x%04x), loginfo(0x%08x)\n", ioc->name,
+			    __func__, ioc_status,
+			    le32_to_cpu(mpi_reply.IOCLogInfo));
+			break;
+		}
+		handle = le16_to_cpu(pcie_device_pg0.DevHandle);
+		device_info = le32_to_cpu(pcie_device_pg0.DeviceInfo);
+		if (!(_scsih_is_nvme_device(device_info)))
+			continue;
+		pcie_device_pg0.WWID = le64_to_cpu(pcie_device_pg0.WWID),
+		pcie_device_pg0.Slot = le16_to_cpu(pcie_device_pg0.Slot);
+		pcie_device_pg0.Flags = le32_to_cpu(pcie_device_pg0.Flags);
+		pcie_device_pg0.DevHandle = handle;
+		_scsih_mark_responding_pcie_device(ioc, &pcie_device_pg0);
+	}
+out:
+	pr_info(MPT3SAS_FMT "search for PCIe end-devices: complete\n",
+	    ioc->name);
+}
+
+/**
  * _scsih_mark_responding_raid_device - mark a raid_device as responding
  * @ioc: per adapter object
  * @wwid: world wide identifier for raid volume
@@ -8931,6 +9077,7 @@ _scsih_scan_for_devices_after_reset(struct MPT3SAS_ADAPTER *ioc)
 {
 	Mpi2ExpanderPage0_t expander_pg0;
 	Mpi2SasDevicePage0_t sas_device_pg0;
+	Mpi26PCIeDevicePage0_t pcie_device_pg0;
 	Mpi2RaidVolPage1_t volume_pg1;
 	Mpi2RaidVolPage0_t volume_pg0;
 	Mpi2RaidPhysDiskPage0_t pd_pg0;
@@ -8941,6 +9088,7 @@ _scsih_scan_for_devices_after_reset(struct MPT3SAS_ADAPTER *ioc)
 	u16 handle, parent_handle;
 	u64 sas_address;
 	struct _sas_device *sas_device;
+	struct _pcie_device *pcie_device;
 	struct _sas_node *expander_device;
 	static struct _raid_device *raid_device;
 	u8 retry_count;
@@ -9166,7 +9314,44 @@ _scsih_scan_for_devices_after_reset(struct MPT3SAS_ADAPTER *ioc)
 	}
 	pr_info(MPT3SAS_FMT "\tscan devices: end devices complete\n",
 	    ioc->name);
+	pr_info(MPT3SAS_FMT "\tscan devices: pcie end devices start\n",
+	    ioc->name);
+
+	/* pcie devices */
+	handle = 0xFFFF;
+	while (!(mpt3sas_config_get_pcie_device_pg0(ioc, &mpi_reply,
+		&pcie_device_pg0, MPI26_PCIE_DEVICE_PGAD_FORM_GET_NEXT_HANDLE,
+		handle))) {
+		ioc_status = le16_to_cpu(mpi_reply.IOCStatus)
+				& MPI2_IOCSTATUS_MASK;
+		if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
+			pr_info(MPT3SAS_FMT "\tbreak from pcie end device"
+				" scan: ioc_status(0x%04x), loginfo(0x%08x)\n",
+				ioc->name, ioc_status,
+				le32_to_cpu(mpi_reply.IOCLogInfo));
+			break;
+		}
+		handle = le16_to_cpu(pcie_device_pg0.DevHandle);
+		if (!(_scsih_is_nvme_device(
+			le32_to_cpu(pcie_device_pg0.DeviceInfo))))
+			continue;
+		pcie_device = mpt3sas_get_pdev_by_wwid(ioc,
+				le64_to_cpu(pcie_device_pg0.WWID));
+		if (pcie_device) {
+			pcie_device_put(pcie_device);
+			continue;
+		}
+		retry_count = 0;
+		parent_handle = le16_to_cpu(pcie_device_pg0.ParentDevHandle);
+		_scsih_pcie_add_device(ioc, handle);
 
+		pr_info(MPT3SAS_FMT "\tAFTER adding pcie end device: "
+			"handle (0x%04x), wwid(0x%016llx)\n", ioc->name,
+			handle,
+			(unsigned long long) le64_to_cpu(pcie_device_pg0.WWID));
+	}
+	pr_info(MPT3SAS_FMT "\tpcie devices: pcie end devices complete\n",
+		ioc->name);
 	pr_info(MPT3SAS_FMT "scan devices: complete\n", ioc->name);
 }
 /**
@@ -9216,6 +9401,7 @@ mpt3sas_scsih_reset_handler(struct MPT3SAS_ADAPTER *ioc, int reset_phase)
 		    !ioc->sas_hba.num_phys)) {
 			_scsih_prep_device_scan(ioc);
 			_scsih_search_responding_sas_devices(ioc);
+			_scsih_search_responding_pcie_devices(ioc);
 			_scsih_search_responding_raid_devices(ioc);
 			_scsih_search_responding_expanders(ioc);
 			_scsih_error_recovery_delete_devices(ioc);
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:35 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:35 +0530
Subject: [PATCH 09/13] mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-10-git-send-email-suganath-prabu.subramani@broadcom.com>

Added debug information for NVMe/PCIe drives in target rest path

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 80 ++++++++++++++++++++++++++++++++----
 1 file changed, 71 insertions(+), 9 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 81f4d2d..dc1fdad 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2929,6 +2929,7 @@ _scsih_tm_display_info(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd)
 	struct scsi_target *starget = scmd->device->sdev_target;
 	struct MPT3SAS_TARGET *priv_target = starget->hostdata;
 	struct _sas_device *sas_device = NULL;
+	struct _pcie_device *pcie_device = NULL;
 	unsigned long flags;
 	char *device_str = NULL;
 
@@ -2945,6 +2946,31 @@ _scsih_tm_display_info(struct MPT3SAS_ADAPTER *ioc, struct scsi_cmnd *scmd)
 			"%s handle(0x%04x), %s wwid(0x%016llx)\n",
 			device_str, priv_target->handle,
 		    device_str, (unsigned long long)priv_target->sas_address);
+
+	} else if (priv_target->flags & MPT_TARGET_FLAGS_PCIE_DEVICE) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_from_target(ioc, priv_target);
+		if (pcie_device) {
+			starget_printk(KERN_INFO, starget,
+				"handle(0x%04x), wwid(0x%016llx), port(%d)\n",
+				pcie_device->handle,
+				(unsigned long long)pcie_device->wwid,
+				pcie_device->port_num);
+			if (pcie_device->enclosure_handle != 0)
+				starget_printk(KERN_INFO, starget,
+					"enclosure logical id(0x%016llx), slot(%d)\n",
+					(unsigned long long)
+					pcie_device->enclosure_logical_id,
+					pcie_device->slot);
+			if (pcie_device->connector_name[0] != '\0')
+				starget_printk(KERN_INFO, starget,
+					"enclosure level(0x%04x), connector name( %s)\n",
+					pcie_device->enclosure_level,
+					pcie_device->connector_name);
+			pcie_device_put(pcie_device);
+		}
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+
 	} else {
 		spin_lock_irqsave(&ioc->sas_device_lock, flags);
 		sas_device = __mpt3sas_get_sdev_from_target(ioc, priv_target);
@@ -3680,18 +3706,14 @@ _scsih_tm_tr_send(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 	Mpi2SCSITaskManagementRequest_t *mpi_request;
 	u16 smid;
 	struct _sas_device *sas_device = NULL;
+	struct _pcie_device *pcie_device = NULL;
 	struct MPT3SAS_TARGET *sas_target_priv_data = NULL;
 	u64 sas_address = 0;
 	unsigned long flags;
 	struct _tr_list *delayed_tr;
 	u32 ioc_state;
 
-	if (ioc->remove_host) {
-		dewtprintk(ioc, pr_info(MPT3SAS_FMT
-			"%s: host has been removed: handle(0x%04x)\n",
-			__func__, ioc->name, handle));
-		return;
-	} else if (ioc->pci_error_recovery) {
+	if (ioc->pci_error_recovery) {
 		dewtprintk(ioc, pr_info(MPT3SAS_FMT
 			"%s: host in pci error recovery: handle(0x%04x)\n",
 			__func__, ioc->name,
@@ -3722,14 +3744,52 @@ _scsih_tm_tr_send(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 		sas_address = sas_device->sas_address;
 	}
 	spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
-
+	if (!sas_device) {
+		spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+		pcie_device = __mpt3sas_get_pdev_by_handle(ioc, handle);
+		if (pcie_device && pcie_device->starget &&
+			pcie_device->starget->hostdata) {
+			sas_target_priv_data = pcie_device->starget->hostdata;
+			sas_target_priv_data->deleted = 1;
+			sas_address = pcie_device->wwid;
+		}
+		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+	}
 	if (sas_target_priv_data) {
 		dewtprintk(ioc, pr_info(MPT3SAS_FMT
 			"setting delete flag: handle(0x%04x), sas_addr(0x%016llx)\n",
 			ioc->name, handle,
 		    (unsigned long long)sas_address));
-		dewtprintk(ioc, _scsih_display_enclosure_chassis_info(ioc,
-		    sas_device, NULL, NULL));
+		if (sas_device) {
+			if (sas_device->enclosure_handle != 0)
+				dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				    "setting delete flag:enclosure logical "
+				    "id(0x%016llx), slot(%d)\n", ioc->name,
+				    (unsigned long long)
+				    sas_device->enclosure_logical_id,
+				    sas_device->slot));
+			if (sas_device->connector_name[0] != '\0')
+				dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				    "setting delete flag: enclosure "
+				    "level(0x%04x), connector name( %s)\n",
+				    ioc->name, sas_device->enclosure_level,
+				    sas_device->connector_name));
+		} else if (pcie_device) {
+			if (pcie_device->enclosure_handle != 0)
+				dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				    "setting delete flag: logical "
+				    "id(0x%016llx), slot(%d)\n", ioc->name,
+				    (unsigned long long)
+				    pcie_device->enclosure_logical_id,
+				    pcie_device->slot));
+			if (pcie_device->connector_name[0] != '\0')
+				dewtprintk(ioc, pr_info(MPT3SAS_FMT
+				    "setting delete flag:, enclosure "
+				    "level(0x%04x), "
+				    "connector name( %s)\n", ioc->name,
+				    pcie_device->enclosure_level,
+				    pcie_device->connector_name));
+		}
 		_scsih_ublock_io_device(ioc, sas_address);
 		sas_target_priv_data->handle = MPT3SAS_INVALID_DEVICE_HANDLE;
 	}
@@ -3764,6 +3824,8 @@ _scsih_tm_tr_send(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 out:
 	if (sas_device)
 		sas_device_put(sas_device);
+	if (pcie_device)
+		pcie_device_put(pcie_device);
 }
 
 /**
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:36 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:36 +0530
Subject: [PATCH 10/13] mpt3sas: NVMe drive support for BTDHMAPPING ioctl
 command and log info
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-11-git-send-email-suganath-prabu.subramani@broadcom.com>

* Added debug prints for pcie devices in ioctl debug path. Which
will be helpful for debugging.
* Added PCIe device support for ioctl BTDHMAPPING ioctl.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_ctl.c | 88 +++++++++++++++++++++++++-------------
 1 file changed, 58 insertions(+), 30 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_ctl.c b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
index 205c443..b4c374b 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_ctl.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_ctl.c
@@ -79,32 +79,6 @@ enum block_state {
 };
 
 /**
- * _ctl_sas_device_find_by_handle - sas device search
- * @ioc: per adapter object
- * @handle: sas device handle (assigned by firmware)
- * Context: Calling function should acquire ioc->sas_device_lock
- *
- * This searches for sas_device based on sas_address, then return sas_device
- * object.
- */
-static struct _sas_device *
-_ctl_sas_device_find_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
-{
-	struct _sas_device *sas_device, *r;
-
-	r = NULL;
-	list_for_each_entry(sas_device, &ioc->sas_device_list, list) {
-		if (sas_device->handle != handle)
-			continue;
-		r = sas_device;
-		goto out;
-	}
-
- out:
-	return r;
-}
-
-/**
  * _ctl_display_some_debug - debug routine
  * @ioc: per adapter object
  * @smid: system request message index
@@ -229,10 +203,9 @@ _ctl_display_some_debug(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 		Mpi2SCSIIOReply_t *scsi_reply =
 		    (Mpi2SCSIIOReply_t *)mpi_reply;
 		struct _sas_device *sas_device = NULL;
-		unsigned long flags;
+		struct _pcie_device *pcie_device = NULL;
 
-		spin_lock_irqsave(&ioc->sas_device_lock, flags);
-		sas_device = _ctl_sas_device_find_by_handle(ioc,
+		sas_device = mpt3sas_get_sdev_by_handle(ioc,
 		    le16_to_cpu(scsi_reply->DevHandle));
 		if (sas_device) {
 			pr_warn(MPT3SAS_FMT "\tsas_address(0x%016llx), phy(%d)\n",
@@ -242,8 +215,25 @@ _ctl_display_some_debug(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 			    "\tenclosure_logical_id(0x%016llx), slot(%d)\n",
 			    ioc->name, (unsigned long long)
 			    sas_device->enclosure_logical_id, sas_device->slot);
+			sas_device_put(sas_device);
+		}
+		if (!sas_device) {
+			pcie_device = mpt3sas_get_pdev_by_handle(ioc,
+				le16_to_cpu(scsi_reply->DevHandle));
+			if (pcie_device) {
+				pr_warn(MPT3SAS_FMT
+				    "\tWWID(0x%016llx), port(%d)\n", ioc->name,
+				    (unsigned long long)pcie_device->wwid,
+				    pcie_device->port_num);
+				if (pcie_device->enclosure_handle != 0)
+					pr_warn(MPT3SAS_FMT
+					    "\tenclosure_logical_id(0x%016llx), slot(%d)\n",
+					    ioc->name, (unsigned long long)
+					    pcie_device->enclosure_logical_id,
+					    pcie_device->slot);
+				pcie_device_put(pcie_device);
+			}
 		}
-		spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
 		if (scsi_reply->SCSIState || scsi_reply->SCSIStatus)
 			pr_info(MPT3SAS_FMT
 			    "\tscsi_state(0x%02x), scsi_status"
@@ -1347,6 +1337,42 @@ _ctl_btdh_search_sas_device(struct MPT3SAS_ADAPTER *ioc,
 }
 
 /**
+ * _ctl_btdh_search_pcie_device - searching for pcie device
+ * @ioc: per adapter object
+ * @btdh: btdh ioctl payload
+ */
+static int
+_ctl_btdh_search_pcie_device(struct MPT3SAS_ADAPTER *ioc,
+	struct mpt3_ioctl_btdh_mapping *btdh)
+{
+	struct _pcie_device *pcie_device;
+	unsigned long flags;
+	int rc = 0;
+
+	if (list_empty(&ioc->pcie_device_list))
+		return rc;
+
+	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
+	list_for_each_entry(pcie_device, &ioc->pcie_device_list, list) {
+		if (btdh->bus == 0xFFFFFFFF && btdh->id == 0xFFFFFFFF &&
+			   btdh->handle == pcie_device->handle) {
+			btdh->bus = pcie_device->channel;
+			btdh->id = pcie_device->id;
+			rc = 1;
+			goto out;
+		} else if (btdh->bus == pcie_device->channel && btdh->id ==
+			   pcie_device->id && btdh->handle == 0xFFFF) {
+			btdh->handle = pcie_device->handle;
+			rc = 1;
+			goto out;
+		}
+	}
+ out:
+	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
+	return rc;
+}
+
+/**
  * _ctl_btdh_search_raid_device - searching for raid device
  * @ioc: per adapter object
  * @btdh: btdh ioctl payload
@@ -1404,6 +1430,8 @@ _ctl_btdh_mapping(struct MPT3SAS_ADAPTER *ioc, void __user *arg)
 
 	rc = _ctl_btdh_search_sas_device(ioc, &karg);
 	if (!rc)
+		rc = _ctl_btdh_search_pcie_device(ioc, &karg);
+	if (!rc)
 		_ctl_btdh_search_raid_device(ioc, &karg);
 
 	if (copy_to_user(arg, &karg, sizeof(karg))) {
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:37 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:37 +0530
Subject: [PATCH 11/13] mpt3sas: Fix nvme drives checking for tlr.
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-12-git-send-email-suganath-prabu.subramani@broadcom.com>

Check for NVMe drives before enabling or checking tlr.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index dc1fdad..72dd57f 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2048,6 +2048,14 @@ scsih_is_raid(struct device *dev)
 	return (sdev->channel == RAID_CHANNEL) ? 1 : 0;
 }
 
+static int
+scsih_is_nvme(struct device *dev)
+{
+	struct scsi_device *sdev = to_scsi_device(dev);
+
+	return (sdev->channel == PCIE_CHANNEL) ? 1 : 0;
+}
+
 /**
  * scsih_get_resync - get raid volume resync percent complete
  * @dev the device struct object
@@ -4834,8 +4842,9 @@ scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd)
 	/* Make sure Device is not raid volume.
 	 * We do not expose raid functionality to upper layer for warpdrive.
 	 */
-	if (!ioc->is_warpdrive && !scsih_is_raid(&scmd->device->sdev_gendev)
-	    && sas_is_tlr_enabled(scmd->device) && scmd->cmd_len != 32)
+	if (((!ioc->is_warpdrive && !scsih_is_raid(&scmd->device->sdev_gendev))
+		&& !scsih_is_nvme(&scmd->device->sdev_gendev))
+		&& sas_is_tlr_enabled(scmd->device) && scmd->cmd_len != 32)
 		mpi_control |= MPI2_SCSIIO_CONTROL_TLR_ON;
 
 	smid = mpt3sas_base_get_smid_scsiio(ioc, ioc->scsi_io_cb_idx, scmd);
@@ -4880,8 +4889,8 @@ scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd)
 
 	raid_device = sas_target_priv_data->raid_device;
 	if (raid_device && raid_device->direct_io_enabled)
-		mpt3sas_setup_direct_io(ioc, scmd, raid_device, mpi_request,
-		    smid);
+		mpt3sas_setup_direct_io(ioc, scmd,
+			raid_device, mpi_request, smid);
 
 	if (likely(mpi_request->Function == MPI2_FUNCTION_SCSI_IO_REQUEST)) {
 		if (sas_target_priv_data->flags & MPT_TARGET_FASTPATH_IO) {
@@ -5411,9 +5420,10 @@ _scsih_io_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index, u32 reply)
 		    le32_to_cpu(mpi_reply->ResponseInfo) & 0xFF;
 	if (!sas_device_priv_data->tlr_snoop_check) {
 		sas_device_priv_data->tlr_snoop_check++;
-		if (!ioc->is_warpdrive &&
+		if ((!ioc->is_warpdrive &&
 		    !scsih_is_raid(&scmd->device->sdev_gendev) &&
-		    sas_is_tlr_enabled(scmd->device) &&
+		    !scsih_is_nvme(&scmd->device->sdev_gendev))
+		    && sas_is_tlr_enabled(scmd->device) &&
 		    response_code == MPI2_SCSITASKMGMT_RSP_INVALID_FRAME) {
 			sas_disable_tlr(scmd->device);
 			sdev_printk(KERN_INFO, scmd->device, "TLR disabled\n");
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:39 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:39 +0530
Subject: [PATCH 13/13] mpt3sas: Update mpt3sas driver version.
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-14-git-send-email-suganath-prabu.subramani@broadcom.com>

Updated mpt3sas driver version to 17.100.00.00

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 2d005af..ef7d001 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -74,8 +74,8 @@
 #define MPT3SAS_DRIVER_NAME		"mpt3sas"
 #define MPT3SAS_AUTHOR "Avago Technologies <MPT-FusionLinux.pdl at avagotech.com>"
 #define MPT3SAS_DESCRIPTION	"LSI MPT Fusion SAS 3.0 Device Driver"
-#define MPT3SAS_DRIVER_VERSION		"16.100.00.00"
-#define MPT3SAS_MAJOR_VERSION		16
+#define MPT3SAS_DRIVER_VERSION		"17.100.00.00"
+#define MPT3SAS_MAJOR_VERSION		17
 #define MPT3SAS_MINOR_VERSION		100
 #define MPT3SAS_BUILD_VERSION		0
 #define MPT3SAS_RELEASE_VERSION	00
-- 
2.5.5



From suganath-prabu.subramani at broadcom.com  Tue Oct 31 05:32:38 2017
From: suganath-prabu.subramani at broadcom.com (Suganath Prabu S)
Date: Tue, 31 Oct 2017 18:02:38 +0530
Subject: [PATCH 12/13] mpt3sas: Fix sparse warnings
In-Reply-To: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
References: <1509453159-7028-1-git-send-email-suganath-prabu.subramani@broadcom.com>
Message-ID: <1509453159-7028-13-git-send-email-suganath-prabu.subramani@broadcom.com>

1) Used variable __le64/__le32 whichever required in
building NVME PRP, and passed to LE Controller.
2) Remove unused functions, And Declared functions as
static which are used only in mpt3sas_scsih.c.

Signed-off-by: Chaitra P B <chaitra.basappa at broadcom.com>
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani at broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c  | 22 ++++++++++-----------
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 37 +++++-------------------------------
 2 files changed, 16 insertions(+), 43 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 0da639d..3061c17 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -1437,8 +1437,8 @@ _base_build_nvme_prp(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 	size_t data_in_sz)
 {
 	int		prp_size = NVME_PRP_SIZE;
-	u64		*prp_entry, *prp1_entry, *prp2_entry, *prp_entry_phys;
-	u64		*prp_page, *prp_page_phys;
+	__le64		*prp_entry, *prp1_entry, *prp2_entry, *prp_entry_phys;
+	__le64		*prp_page, *prp_page_phys;
 	u32		offset, entry_len;
 	u32		page_mask_result, page_mask;
 	dma_addr_t	paddr;
@@ -1455,17 +1455,17 @@ _base_build_nvme_prp(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 	 * PRP1 is located at a 24 byte offset from the start of the NVMe
 	 * command.  Then set the current PRP entry pointer to PRP1.
 	 */
-	prp1_entry = (u64 *)(nvme_encap_request->NVMe_Command +
+	prp1_entry = (__le64 *)(nvme_encap_request->NVMe_Command +
 	    NVME_CMD_PRP1_OFFSET);
-	prp2_entry = (u64 *)(nvme_encap_request->NVMe_Command +
+	prp2_entry = (__le64 *)(nvme_encap_request->NVMe_Command +
 	    NVME_CMD_PRP2_OFFSET);
 	prp_entry = prp1_entry;
 	/*
 	 * For the PRP entries, use the specially allocated buffer of
 	 * contiguous memory.
 	 */
-	prp_page = (u64 *)mpt3sas_base_get_pcie_sgl(ioc, smid);
-	prp_page_phys = (u64 *)mpt3sas_base_get_pcie_sgl_dma(ioc, smid);
+	prp_page = (__le64 *)mpt3sas_base_get_pcie_sgl(ioc, smid);
+	prp_page_phys = (__le64 *)mpt3sas_base_get_pcie_sgl_dma(ioc, smid);
 
 	/*
 	 * Check if we are within 1 entry of a page boundary we don't
@@ -1475,8 +1475,8 @@ _base_build_nvme_prp(struct MPT3SAS_ADAPTER *ioc, u16 smid,
 	page_mask_result = (uintptr_t)((u8 *)prp_page + prp_size) & page_mask;
 	if (!page_mask_result) {
 		/* Bump up to next page boundary. */
-		prp_page = (u64 *)((u8 *)prp_page + prp_size);
-		prp_page_phys = (u64 *)((u8 *)prp_page_phys + prp_size);
+		prp_page = (__le64 *)((u8 *)prp_page + prp_size);
+		prp_page_phys = (__le64 *)((u8 *)prp_page_phys + prp_size);
 	}
 
 	/*
@@ -1604,7 +1604,7 @@ _base_build_nvme_prp(struct MPT3SAS_ADAPTER *ioc, u16 smid,
  * Returns:		true: PRPs are built
  *			false: IEEE SGLs needs to be built
  */
-void
+static void
 base_make_prp_nvme(struct MPT3SAS_ADAPTER *ioc,
 		struct scsi_cmnd *scmd,
 		Mpi25SCSIIORequest_t *mpi_request,
@@ -1612,7 +1612,7 @@ base_make_prp_nvme(struct MPT3SAS_ADAPTER *ioc,
 {
 	int sge_len, offset, num_prp_in_chain = 0;
 	Mpi25IeeeSgeChain64_t *main_chain_element, *ptr_first_sgl;
-	u64 *curr_buff;
+	__le64 *curr_buff;
 	dma_addr_t msg_phys;
 	u64 sge_addr;
 	u32 page_mask, page_mask_result;
@@ -1740,7 +1740,7 @@ base_is_prp_possible(struct MPT3SAS_ADAPTER *ioc,
 	struct scatterlist *sg_scmd;
 	bool build_prp = true;
 
-	data_length = cpu_to_le32(scsi_bufflen(scmd));
+	data_length = scsi_bufflen(scmd);
 	sg_scmd = scsi_sglist(scmd);
 
 	/* If Datalenth is <= 16K and number of SGE???s entries are <= 2
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 72dd57f..31e9343 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -599,7 +599,7 @@ __mpt3sas_get_pdev_from_target(struct MPT3SAS_ADAPTER *ioc,
  *
  * This searches for pcie_device from target, then return pcie_device object.
  */
-struct _pcie_device *
+static struct _pcie_device *
 mpt3sas_get_pdev_from_target(struct MPT3SAS_ADAPTER *ioc,
 	struct MPT3SAS_TARGET *tgt_priv)
 {
@@ -942,7 +942,7 @@ _scsih_sas_device_init_add(struct MPT3SAS_ADAPTER *ioc,
 }
 
 
-struct _pcie_device *
+static struct _pcie_device *
 __mpt3sas_get_pdev_by_wwid(struct MPT3SAS_ADAPTER *ioc, u64 wwid)
 {
 	struct _pcie_device *pcie_device;
@@ -975,7 +975,7 @@ found_device:
  *
  * This searches for pcie_device based on wwid, then return pcie_device object.
  */
-struct _pcie_device *
+static struct _pcie_device *
 mpt3sas_get_pdev_by_wwid(struct MPT3SAS_ADAPTER *ioc, u64 wwid)
 {
 	struct _pcie_device *pcie_device;
@@ -989,7 +989,7 @@ mpt3sas_get_pdev_by_wwid(struct MPT3SAS_ADAPTER *ioc, u64 wwid)
 }
 
 
-struct _pcie_device *
+static struct _pcie_device *
 __mpt3sas_get_pdev_by_idchannel(struct MPT3SAS_ADAPTER *ioc, int id,
 	int channel)
 {
@@ -1012,34 +1012,7 @@ found_device:
 	return pcie_device;
 }
 
-
-/**
- * mpt3sas_get_pdev_by_idchannel - pcie device search
- * @ioc: per adapter object
- * @id: Target ID
- * @channel: Channel ID
- *
- * Context: This function will acquire ioc->pcie_device_lock and will release
- * before returning the pcie_device object.
- *
- * This searches for pcie_device based on id and channel, then return
- * pcie_device object.
- */
-struct _pcie_device *
-mpt3sas_get_pdev_by_idchannel(struct MPT3SAS_ADAPTER *ioc, int id, int channel)
-{
-	struct _pcie_device *pcie_device;
-	unsigned long flags;
-
-	spin_lock_irqsave(&ioc->pcie_device_lock, flags);
-	pcie_device = __mpt3sas_get_pdev_by_idchannel(ioc, id, channel);
-	spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
-
-	return pcie_device;
-}
-
-
-struct _pcie_device *
+static struct _pcie_device *
 __mpt3sas_get_pdev_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 {
 	struct _pcie_device *pcie_device;
-- 
2.5.5



